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(57) An Image generating apparatus comprising: a 
camera or a plurality of cameras; space reconstructing 
means for mapping the input Image from said camera, 
into a predetermined space model of a predetermined 
three-dimensional space; point-of-view converting 
means of synthesizing an Image viewed from an arbi- 
trary virtual point of view In said predetennined three- 
dimensional space by referencing to said space data 
mapped by said space reconstmcting means; and dis- 
playing means of displaying the image converted by 
said point-of-vlew converting means. 
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Description 
Technical Reld 

5 [0001] The present invention relates to an apparatus and a method for displaying a synthesized single image conv 
posed of a plurality of images shot by plural cameras, instead of displaying the images independently, so that the 
overall situation of the area shot by said plural cameras can be intuitlonally understood, and for example, to a monitoring 
apparatus in a shop or a vehicle's surroundings monitoring apparatus, etc. for the assistance of the safety confirmation 
during the driving of a vehicle. 

10 

Background Art 

[0002] A prior art general nrK}nitoring apparatus Is generally configured so that a monitoring objective part in a shop 
and the like is shot by a camera or plural cameras and that the image(s) is displayed on a monitor screen. In this case, 

IS for example, when plural cameras are installed, the same number of monitor screens as the number of said cameras 
are generally prepared. When ttie same number of monitor screens as the number of the cameras can not be prepared, 
the images of said plural cameras are Integrated into a single image using a separator and the like, the camera Images 
are sequentially switched, or another method is used. However, there has been such a problem or the like that a 
manager needs to consider the continuity between the Independently displayed images during the monitoring of the 

20 image of each camera in such a prior art apparatus. 

[0003] To resolve the problem, for example, Japanese Laid-Open Patent Publteation No. Hei 10-164566, etc. dis- 
closes a monitoring apparatus in which a plurality of cameras are installed so that the images thereof overiap at the 
edge parts and in which the Images of the plurality of cameras are Integrated and displayed into a single image by 
superposing said overiap'ping parts, thereby resolving said problem of continuity. 

25 [0004] Another example of the appttoation of a monitoring apparatus is the case of the installation on a vehrcle. One 
of the prior art example is as follows. That is a monitoring apparatus In which a camera is installed for monitoring the 
surroundings of a vehicle and the image acquired by said camera is displayed on a TV monitor installed near the 
driver's seat. An apparatus is known in which a place, such as the rear side of the vehtele, unseen to the driver with 
the own eyes or a mirror is shot by said camera and displayed on said monitor. In particular, many are used on a 

30 vehicle, such as a large truck and a van (one-box wagon), having a nan^ow field of view seen with the eyes and a min-or. 
[0005] Figure 69 shows a prior art example.in which monitoring cameras are installed on a vehicle. The example of 
Figure 69 is a method in which the images of four monitoring cameras (C1-C4) mounted on the body of the vehicle 
are synthesized into a single image through a separating adapter and the Images are separately displayed on a TV 
monitor (D1-D4). In particular, with regard to the image of the rear side camera, an invention is made so that the right 

35 and left of the image is reversed in order that the positional relation of the objects viewed in the Image becomes the 
same as viewed in a mirror. Further, with regard to a place unseen due to the limitation in the angle of field of the 
camera, an invention is made so that each camera is turned by manual operation by the driver thereby pennitting to 
obtain the image of the desired place. Such a monitoring apparatus mentioned above is disclosed, for example, in 
Japanese Laid-Open Patent Publteation No. Hei 5-^10078, and so on. 

40 [0006] However, in a prior art monitoring apparatus such as the above-mentioned example, the images of respective 
camera s are displayed independently of each other. Accordingly, it is difficult to understand instantaneously the overall ' 
situation of the space shot by the cameras by watching each Image. 

[0007] In addition, In case of synthesizing the respective images Into a single image, the apparatus cannot treat, for 
example, a request by the manager to watch an integrated image viewed from a desired point of view depending on 

45 the situation, because the apparatus does not calculate the spatial posltk>n of an object shot by the camera. 

[0008] The same problem occurs in a monitoring apparatus installed on a vehicle. That is, in a prior ari monitoring 
apparatus such as the above-mentioned example, the images Input from respective cameras are displayed independ- 
ently of each other. Accordingly, for example, in case of using said displayed images as assistance during the parking 
of a vehble in a parking space, said images merely serve to provide a view othenmse unseen. That is, the only effect 

50 is to reduce the area unseen to the driver in comparison with the conventional case by visual inspection and a minror. 
[0009] Meanwhile, a wide-angle lens is generally used in order to overcome the narrowness of the field. The image 
from the wide-angle lens does not provide the detail in a specif b part. However, the overall situation of the surroundings 
of the vehicle is easily understood because of the wldeness of the field of the wide-angle tens. 
[0010] However, even if an ordinary lens is replaced by the wide-angle lens, the image of the camera still depends 

55 on the position on the body of a vehk^le where the camera is installed, and accordingly, it is not possible to obtain a 
virtual Image viewed from a position where the camera is not installed. That is, the only effect of using the camera 
equipped with the wide-angle lens is to reduce the number of the cameras. 

[001 1] Another prior art apparatus is further disclosed in Japanese Laid-Open Patent Publication No. Hei 3-99952. 
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Rgure 70 is a block diagram showing an embodinnent of the prior art apparatus for monitoring the surroundings of a 
vehicle. The images input from cameras 1-N 2201 into an image converting sec^on 2202 are converted into another 
coordinates by a conversion, and synthesized into a single image by an image displaying section 2203. It is then 
displayed on a TV monitor 2204 installed at the driver's seat. In the image displaying section, an invention is possible 
5 in which the displayed position of the vehicle in question is shifted from the center of the screen or the llice. depending 
on gear position, vehicle speed, and winker operation in order to widen the area of interest of the surroundings of the 
vehicle, and the like. 

[0012] Further another prior art example developed further is disclosed in Japanese Lald-Open Patent Publication 
No. Hei 7-186833. In this example, in the presentation of the situation of the surroundings to the driver, road surface 

10 part is distinguished from the other part. Then, the road surface part is converted into an image viewed downward from 
the point of view above the vehicle center by a coordinate transformation. The part other than the road surface of the 
raw video from the camera is altered property with regard to place and size, and displayed in superposition on said 
converted image. As such, it accurately inf omis the situation of an ot)stacle in the sun^oundings of the vehicle, especially 
another vehicle and the like approaching from behind the vehk:le. 

15 [0013] However, in the above-mentioned another prior art apparatus for monitoring the surroundings of a vehble, 
there has been a problem that it can be difficult to find the con-espondence of the obtained synthesized image to the 
actual object. 

[0014] In the above-mentioned further another example, by separating the road surface part and the part other than 
the road surface, the viewed image of the objects outside of the road surface is cut out and pasted into a synthesized 
20 Image. However, the cuttlng-out of the objects outside of the road surface is one of the diffbult problems in the field of 
image recognition, and hence difficult to realize practically. 

[0015] on the other hand, there is a problem that the shift of the camera during the running causes a shift in the 
synthesized image. A method for resolving this is disclosed, for example, in Japanese Laid-Open Patent Publicatton 
No. Hei 5-310078. In this method, a mechanism for changing the orientation of a camera is provided, whereby the 
25 driver manually directs the camera into a desired direction while viewing a synthesized image. The problem in this 
case Is, however, that the mechanisms for changing the orientation are necessary to installed in the same number as 
the number of the cameras, which causes a higher cost. 

[001 6] To resoh^e such problems, in case of an apparatus installed on a vehble for example, an object of the present 
Invention is to compose a single image showing what objects are present near the vehicle in the all sun'oundings of 
30 the vehicle, as realistically and understandably as possible, thereby displaying it to the driver. 

[0017] The present Invention further provides a method for easily obtaining camera parameters, such as the attach- 
ment position and the attachment angle of the camera, and also an apparatus and a method for detecting and connecting 
the shift of said camera parameters due to vibration, temperature and the like, if it occurs during the running. 

35 Disclosure of I nventk)n 

[0018] To solve the above-described problems, the present invention is an image generating apparatus comprising: 
a camera or a plurality of cameras; space reconstructing means for mapping the input image from said camera, into 
a predetermined space model of a predetermined three-dimensional space; point-of-view converting means for syn- 
40 thesizing an image viewed from an arbitrary virtual point of view in said predetermined three-dimensional space by 
referencing to said space data mapped by said space reconstructing means; and displaying means for displaying the 
image converted by said point-of-view converting means. 

[0019] Further, to solve the above-described problems, an Image generating method of the present invention com- 
prises: a space reconstructing step of mapping the Input image from a camera locating In a predetermined three- 
45 dimensional space, into a predetennlned space model of said predetemnined three^imensional space; and a point- 
of-view converting step of synthesizing an image viewed from an arbitrary virtual point of view in said predetemnined 
three-dimensional space by referencing to said space data. 

[0020] The baste configuratron of the present invention is characterized by comprising: image inputting means of 
inputting an image or images from a camera or a plurality of cameras installed on a vehicle; a camera parameter table 

50 for storing the camera parameters indicating the characteristics of said camera or cameras; space model generating 
means of generating a space model in a coordinate system with the reference on the vehicle; mapping means of 
mapping the image input from said camera, into said space model; point-of-view converting means of setting a point 
of view and synthesizing a single image viewed from said point of view using the data generated by said mapping 
means; and displaying means of displaying the image converted by said point-of-view converting means. 

55 [0021 ] The first applied configuration of a monitoring apparatus of the present invention is characterized by compris- 
ing: a distance sensor for measuring the distance; and obstacle detecting means for measuring, as the situation of the 
surroundings of a vehicle, at least the distance to an obstacle existing in the sun^oundings of the vehicle by said distance 
sensor. 
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[0022] In an apparatus in accordance with the present invention, the space model generating means generates a 
model previously set properly or a model set depending on the distance of an obstacle in the sun^oundings of the vehicle 
sensed tiy an obstacle sensing means. The mapping means maps, into said space model, the Image of the sunroundlngs 
of the vehicle input by the image inputting means from the camera installed on the vehicle. Then, the point-of-view 
s converting means composes a single image viewed from a predetemnined point of view using the mapped image, and 
the displaying means displays it. Here, the crew of the vehicle can cause displaying the image viewed from a desired 
point of view. 

[0023] In order to resoh^e the above-mentioned problems, the basb configuration of the present invention is char- 
acterized by comprising: image inputting means of inputting an image or images from a camera or a plurality of cameras 

w installed on a vehicle; a camera parameter table for storing the camera parameters indicating the characteristics of 
said camera or cameras; road-surface characteristics detecting means of detecting the characteristics on the road 
surface as the situation of the surroundings of the vehicle; space model generating means of generating a space model 
corresponding to the process result of said road-surface characteristics detecting means in a coordinate system with 
the reference on the vehicle; mapping means of mapping the image input from said camera, into said space model; 

IS point-of-view converting means of setting a point of view and synthesizing a single image viewed from said point of 
view using the data generated by said mapping means; and displaying means of displaying the image converted by 
said point-of-view converting means. 

[0024] The first applied configuration of the present invention is characterized by further comprising: movement di- 
rection detecting means for detecting the movement direction of said vehicle; and movement distance detecting means 

20 for detecting the movement distance of the vehicle in a unit time; and wherein said apparatus successively corrects 
said space model depending on the present position of the characteristics on the road surface obtained from the 
process results of said movement direction detecting means and said movement distance detecting means. 
[0025] The second applied configuration of the present Invention is characterized by comprising characteristics cor- 
recting means for connecting the process result white displaying the process result of said road-surface characteristics 

25 detecting means on said displaying means. 

[0026] In the present invention, the road-surface characteristics detecting means detects the characteristics, such 
as a white line, on the road surface, and the space model generating means generates a space model con-espondingly 
to said detected characteristics. The mapping means maps, into said space model, the image of the sunroundlngs of 
the vehicle input by the image inputting means from the camera installed on the vehicle. Then, the point-of-view con- 

30 verting means composes a single image viewed from a predetermined point of view using the mapped image, and the 
displaying means displays it. Since the positional relation between the vehicle and the road surface characteristics 
changes as the vehicle moves, the space model is con-ected in response to the change, and an image is synthesized 
using the comected space model and then displayed. 

[0027] In order to achieve the above-mentioned object, the present invention is characterized by comprising: image 
35 inputting means of inputting an image or images from a camera or a plurality of cameras installed on a vehicle; a 
camera parameter table for storing the camera parameters indicating the characteristics of said camera or cameras; 
mapping nneans of mapping the image input from said camera, into a space model generated by modeling the sur- 
roundings of said vehicle; point-of-view converting means of synthesizing a single image viewed from a desired virtual 
point of view using the data generated by said mapping means; camera parameter correcting means of correcting the 
40 parameters of said camera or cameras for each camera independently; and displaying means of displaying the image 
converted by said point-of-view converting means. 

[0028] Said point-of-view converting means is characterized by switching said virtual point of view according to the 

time of process or the time of normal operation of said camera parameter correcting means. 

[0029] Said point-of-view converting means makes said virtual point of view consistent with the camera parameter 

45 of any of said cameras on a vehicle In the time of process of said camera parameter correcting means. 

[0030] Said point-of-view converting means makes said virtual point of view consistent with the yet-corrected camera 
parameter of the camera being under the connection, in the time of process of said camera parameter correcting means. 
[0031 ] The direction of the operation of changing the direction of the point of view is set to be reverse to the direction 
of said point of view, in said point-of-view converting means. 

50 [0032] Said displaying means displays a marie indbating a boundary, in superposition on the synthesized Image in 
the boundary part where respective images contact when displaying the Image from each camera. 
[0033] In order to achieve the above-mentioned object, the present invention is characterized by cornprlsing: image 
Inputting means of inputting an Image or images from a camera or a plurality of cameras Installed on a vehble; a 
camera parameter table for storing the camera parameters Indbating the characteristics of said camera or cameras; 

55 mapping means of mapping the image input from said camera, into a space model generated by modeling the sur- 
roundings of the vehicle; a point-of-view parameter table for storing the point-of-view parameters including at least the 
position and tiie orientation; point-of-view converting means of synthesizing an image viewed from a desired virtual 
point of view using the result of the mapping process by said mapping means; point-of-view parameter con'ecting 
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means of corre<iing the parameters of said virtual point of view; and displaying means of displaying the image converted 
by said point-of-view converting means. 

[0034] The present invention is characterized by said set of the point-of-view parameters being distinguished so as 
to con-espond to any one of the cameras installed on the vehicle and stored in said poiht-of-view parameter table. 
5 [0035] The direction of operation is set to be reverse to the actual alteration of point-of-view parameter at least for 
the operations in orientation , position, and rotation among the operations of changing said virtual-point-of-view param- 
eter in said point-of-view parameter correcting means. 

[0036] Said point-of-view parameter correcting means correct said virtual-point-of-view paranwter. a fixed tentative 
virtual point of view is provided, and the progress of the con-ection of the virtual point of view being under the con^on 

10 is successively displayed as a synthesized image viewed from said tentative virtual point of view. 

[0037] Said displaying means displays a martc indicating a boundary, In superposition on the synthesized image in 
the boundary part where respective images contact when displaying the image from each camera. 
[0038] Furthermore, the present invention is characterized by comprising a mapping table for retaining the pixel 
correspondence relation between the camera Input image and the synttiesized image, whereby said con^pondence 

15 relation obtained in the processes by said mapping means and said point-of-view converting means is stored in said 
mapping table. 

[0039] Further, the present invention is characterized by comprising mapping table connecting means of re-calculating 
said mapping table using the point-of-view parameters altered in the process by the point-of-view parameter correcting 
means. 

20 

Brief Description of Drawings 

[0040] Figure 1 is a block diagram showing an example of the basic configuration of an image generating apparatus 
in accordance with the present invention (Claim 1). 
25 [0041] Figure 2 is a blocic diagram showing an example of tiie configuration of an image generating apparatus in 
accordance with the combination of the present inventions of Claims 9 and 12. 

[0042] Figure 3 is a block diagram showing an example of the configuration of an image generating apparatus in 
accordance with the present invention (Claim 14). 

[0043] Figure 4 Is a block diagram showing an example of the configuration of an image generating apparatus in 
30 accordance with the present invention (Claim 1 7). 

[0044] Figure 5 is a block diagram showing an example of the configuration of an Image generating apparatus in 
accordance with the present invention (Claim 18). 

[0045] Figure 8 is a block diagram showing an image generating apparatus as tfie integration of Figures 1 to 5. 
[0046] Figure 7 Is a conceptual diagram showing an example of the mounting of a camera onto a vehicle. 
35 [0047] Figure 8 is a conceptual diagram representing the con-espondence relation between a point of the U-V coor- 
dinate system being set on the plane including an image shot by a camera and a point of the three-dimensional space 
coordinate system. 

[0048] Figure 9 is a diagram showing the data stored in a camera parameter table 1 03, In tiie fomn of a table. 
[0049] Figure 1 0 Is a diagram showing an example of a temperature correclton table 111 , In the fomi of a table. 
40 [0050] Figure 1 1 is a diagram showing an example of a camera parameter table 1 03 renewed using a degree cor- 
rection table, in the fomn of a table. 

[0051] Figure 12 is a diagram showing an example of the description of a space data buffer 105 for storing space 
data, in the fomn of a table. 

[0052] Figure 13 is a conceptual diagram of the positional relation between the characteristic points on the road 
45 surface and the vehicle, viewed from above. 

[0053] Figure 1 4 is a conceptual diagram showing an Image of the road surface including said characteristic points, 
shot by the car-canied camera 1 of Figure 13, 

[0054] Figure 1 5 is a conceptual diagram showing an image of the road surface including said characteristk: points, 
shot by the car-canied camera 2 of Figure 13. 
so [0055] Figure 1 6 is a conceptual diagram of the positional relation between characteristic points A, B, C on the road 
surface and the vehrcle, viewed from above. 

[0056] Figure 1 7 is a conceptual diagram showing an image of the road surface including said characteristic points 
A, B, shot by the car-camed camera 1 of Rgure 1 6. 

[0057] Figure 1 8 is a conceptual diagram showing an image of the road surface including said characteristic points 
55 B, C, shot by the car-carried camera 2 of Figure 1 6. 

[0058] Figure 1 9 is a conceptual diagram showing the situation of the synthesizing of an Image viewed from a virtual 
camera, by point-of-view converting means 1 06 in accordance with the present invention using the images shot by the 
car-canied cameras 1 and 2 of Figure 16. 
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[0059] Rgure 20 shows: 

(A) a conceptual diagram showing the case that a camera is instailed ak>ove the approximate center of a vehicle 
downwardly, as an example of the place of Installation of a virtual camera; 

(B) a conceptual diagram showing the case that a camera is Instailed obliquely above the approximate front side 
of a vehicle, direc^ng the camera to the vehicle, as an example of the place of installation of a virtual camera; and 

(C) a conceptual diagram showing an example of the synthesized image of the above-mentioned case(B). 

[0060] Figure 21 shows: 

(a) a conceptual diagram showing the situation that a pattem light irradiating apparatus as characteristic point 
generating means is attached on top of a side body of a vehicle; 

(b) a conceptual diagram showing the situation that plural pattem light irradiating apparatuses are attached on top 
of a body of a vehicle thereby irradiating the pattem light onto the road surface, viewed from top of the vehide; and 

(c) a conceptual diagram showing the situation that the rectangle patterns irradiated onto the road surface are shot 
fay a camera. 

[0061] Figure 22 is a flow chart showing the procedure of the process of renewing a camera parameter table 1 03 

depending on the temperature, in calibrating means 102. 

[0062] Figure 23 is a flow chart showing the procedure of the process in space data converting means 114. 
[0063] Figure 24 is a conceptual diagram used for the assistance of the description of space data converting means 
114. 

[0064] Figure 25 is a flow chart showing the procedure of the process of checking the record of correction history 
thereby issuing a correction indication when necessary. 

[0065] Figure 26 is a flow chart showing the flow of the overall process of an image generating apparatus in accord- 
ance with the present invention. 
[0066] Figure 27 shows: 

(a) a blocic diagram showing an example of the basic configuration of an Image generating apparatus in accordance 
with the present invention (Claim 34); and 

(b) a block diagram showing an example of the configuration of an Image generating apparatus in accordance with 
the present invention (Claim 37). 

[0067] Figures 28(a) to 28(d) are conceptual diagrams showing a space model in a bird's eye view. 
[0068] Figure 29 is a conceptual diagram representing the correspondence relation between a point in the U-V co- 
ordinate system being set In the plane including an image shot by a camera and a point in the three<limensional space 
coordinate system. 

[0069] Figure 30 is a flow chart showing the procedure of the mapping process in mapping means 104A. 
[0070] Figure 31 is a conceptual diagram showing a method of setting a screen plane in the three-dimensional space 
depending on the distance between a vehk^le and an obstacle existing in the surroundings of the vehble. 
[0071] Figure 32 is a flow chart showing the flow of the overall process of an image generating apparatus In accord- 
ance with the present invention. 
[0072] Figure 33 shows: 

(a) a block diagram showing an example of the configuration of an apparatus for monitoring the surroundings of 
a vehicle in accordance with the present invention (Claim 39); 

(b) a block diagram showing an example of the configuration of an apparatus for monitoring the surroundings of 
a vehicle in accordance with the present invention (Claim 41 ); and 

(c) a block diagram showing an example of the configuration of an apparatus for monitoring the surroundings of 
a vehicle in accordance with the present invention (Claim 44). 

[0073] Figure 34 shows: 

(a) a conceptual diagram showing a space model in accordance with the present inventton, in a bird's eye view; and 

(b) a conceptual diagram showing a space model in accordance with the present inventbn, in a perspective pro- 
jection view from above the vehicle downwardly. 

[0074] Figures 35(a) to 35(d) are diagrams showing an example of the process of extracting a characteristk: point 
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by road-surface characteristics detecting means 103B. 

[0075] Figure 36 Is a flow chart showing the flow of the process of extracting a characteristic point. 
[0076] Figure 37 is a flow chart showing the procedure of the process of calculating the position of a characteristic 
point as the vehicle moves, in an apparatus for monitoring the sunroundings of a vehicle in accordance with the present 
invention (Claim 41). 

[0077] Figure 38 is a diagram showing the process in characteristics position calculating means. 

[0078] Figure 39 is a conceptual diagram showing the situation that a characteristics correcting process is displayed 

on displaying means 1076. 

[0079] Figure 40 is a flow chart showing the flow of the process In a characteristics correcting process. 

[0080] Figure41 is af tow chart showing the flow of the overall process of an apparatus for monitoring the sum)undings 

of a vehicle in accordance with the present invention. 

[0081] Figure 42 is a block diagram showing an example of the configuration of an image generating apparatus in 
accordance with the present invention (Claim 45). 
[0082] Figure 43 shows: 

(a) a conc:eptual diagram showing an example of car-carmd cameras installed on a vehicle; and 

(b) a conceptual diagram showing the displayed region on a monitor of the image of each car-carried camera. 

[0083] Figure 44 is a conceptual diagram showing the three-dimensional space coordinate system based on a ve- 
hicle. 

[0084] Figure 45 is a diagram showing the data stored in a camera parameter table 1 02C, In the form of a table. 
[0085] Figure 46 is a diagram showing the data stored in a point-of-view parameter table, in the form of a table. 
[0086] Figure 47 Is a conceptual diagram representing the correspondence relation between a point in the U-V co- 
ordinate system being set in the plane including an image shot by a camera and a point in the three-dimensional space 
coordinate system. 

[0087] Figure 48 is a conceptual diagram showing an example of the configuration of an operating section for cor- 
recting a camera parameter by camera parameter correcting means 1 06C. 

[0088] Figure 49 is a bloctc diagram showing an example of the configuration of an art relevant to an image generating 
apparatus In accordance with the present invention. 

[0089] Figure 50 is a conceptual diagram showing a mapping table 108C, in the form of a table. 
[0090] Figure 51 is a flow chart showing the flow of the process of connecting a camera parameter 
[0091] Figures 52 (a) and 52 (b) are conceptual diagrams showing an example of a display screen during the cor- 
rection of a camera paranneter. 

[0092] Figures 53(a) and 53(b) are conceptual diagrams showing a display screen during the time without the cor- 
rection of a canera parameter (that is, normal state). 

[0093] Figure 54 is a block diagram showing an example of the basic configuration of an image generating apparatus 
in accordance with the present Invention (Claim 51). 

[0094] Figure 55 is a block diagram showing an example of the basic configuration of an image generating apparatus 
in accordance with the present Invention (Claim 54). 

[0095] Figure 56 is a conceptual diagram showing the case that the guide data is generated using a point source of 
light. 

[0096] Figure 57 is a conceptual diagram showing the c:ase that the guide data is generated using an edge line of a 
body of a vehicle. 

[0097] Figure 58 is a conceptual diagram showing the situation that the c^albration is performed by using the guide 
data using a point source of light. 

[0098] Figure 59 is a conceptual diagram showing the situation that the calbration is perfonmed by using the guide 
data using an edge line. 

[0099] Figure 60 is a block diagram showing an example of the configuration of an image generating apparatus in 
accordance with the present invention (Claim 55). 

[01 00] Figure 61 is a diagram showing the data stored in a point-of-view parameter table 1 08D, in the fonm of a table. 
[01 01] Figure 62 is a conceptual diagram showing the situation that the image from a (»r-can1ed camera Is converted 
to an image viewed from a virtual point of view. 

[0102] Figure 63 is a conceptual diagram showing an example of the configuration of an operating section for cor- 
recting a point-of-view parameter by point-of-view parameter correcting means 106D. 

[01 03] Figure 64 is a block diagram showing an example of the configuration of an art relevant to an image generating 
apparatus in accordance with the present invention. 
[01 04] Figure 65 shows: 
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(a) a block diagram showing an exanipie of the configuration of an art relevant to an image generating apparatus 
in €K£ordance with the present invention; 

(b) a conceptual diagram showing an example of a car-carried camera synthesized image before point-of-view 
parameter correction; and 

5 (c) a conceptual diagram showing an example of a car-carried camera synthesized image after point-of>view pa- 

rameter correction 



[0105] Figures 66(a) to 66(c) are conceptual diagrams showing a method of altering a mapping table 109D. 

[0108] Figure 67 is a flow chart showing the flow of the process of con-ecting a point-of-view parameter. 

10 [01 07] Figures 68(a) to 68(c) are conceptual diagrams showing the process of correcting a point-of-view parameter. 

[0108] Figure 69 is a block diagram showing an example of the configuration of a prior art image generating appa- 
ratus. 

[0109] Figure 70 is a block diagram showing an example of the conf^uratk>n of a prior art apparatus for monitoring 
the surroundings of a vehk;le. 



15 



[Legend of Reference Numerals] 



[0110] 



20 


101 


Camera 




102 


Calibrating means 




103 


Camera parameter table 




104 


Space reconstructing means 




105 


Space data buffer 


25 


106 


Polnt-of-view converting means 




107 


Displaying means 




108 


Characteristk: point extracting means 




109 


Characteristb point generating means 




110 


Temperature sensor 


30 


111 


Temperature con-ection table 




112 


Movement direction detecting means 




113 


Movement distance detecting means 




114 


Space data converting means 




115 


Correction history recording means 


35 


116 


Camera correction indicating means 




101A 


Camera 




102A 


Camera parameter table 




103A 


Space model generating means 




104A 


Mapping means 


40 


105A 


Point-of-view converting means 




106A 


Displaying means 




107A 


Distance sensor 




108A 


Obstacle detecting means 




101B 


Camera 


45 


102B 


Camera parameter table 




103B 


Road-surface characteristics detecting means 




1048 


Space model generating means 




1058 


Mapping means 




1068 


Point-of-view converting means 


50 


1078 


Displaying means 




1088 


Movement distance detecting means 




1098 


Movement direction detecting means 




1108 


Characteristrcs correcting means 




101C 


Camera 


55 


102C 


Camera parameter table 




103C 


Space model 




104C 


Mapping means 




105C 


Point-of-view converting means 



8 



EP1 115 250 A1 



106C 


Camera parameter correcting means 


107C 


Displaying means 


108C 


l^apping table 


901 C 


Camera selection button 


902C 


Parallel displacement button 


903C 


Joystick 


904C 


Zoom button 


101D 


Camera 


102D 


Camera parameter table 


103D 


Space model 


104D 


Mapping means 


105D 


Point-of'View converting means 


106D 


Point-of-view parameter connecting means 


107D 


Displaying means 


108D 


Point-of-view parameter table 


109D 


Mapping table 


110D 


Mapping table correcting means 


111D 


Buffer 


1001D 


Camera selection button 


1002D 


Parallel displacement button 


1003D 


Joystick 


1004D 


Zoom button 



Best Mode for Carrying Out the Invention 

25 

[0111] The bask; configuration of the present invention is characterized by comprising: a camera or a plurality of 
cameras; a camera parameter table for storing the camera parameters indicating the characteristk^s of said camera 
or cameras; space reconstructing means of mapping the image input from sakJ camera, into a space model of three- 
dimensional space depending on said camera parameters thereby generating space data; a space data buffer for 
30 temporarily storing the space data generated by said space reconstructing means; point-of-view converting means of 
synthesizing an image viewed from an art^itrary point of view by referencing to said space data; and displaying means 
of displaying the image converted by said point-of-view converting means. 

[0112] A first applk:ation configuration of an image generating apparatus in accordance with the present invention 
is characterized by comprising calibrating means for obtaining the camera parameters indicating the camera charac- 

35 teristics by an input or a calculation. 

[01 13] A second applk:ation configuration of an image generating apparatus in accordance with the present invention 
is ^characterized by comprising: characteristk: point generating means of generating, within the field of view of the 
camera, a plurality of points the three-dimensional coordinates of which are identifiable; and characteristb point ex- 
tracting means of extracting those characteristic points. 

^ . [01 14] A third applk:ation configuration of an image generating apparatus in accordance with the present invention 
is characterized by comprising a temperature sensor and a temperature connection table. 

[01 15] A fourth application configuration of an image generating apparatus in accordance with the present invention 
is characterized by comprising: movement direction detecting means of detecting the direction of the movement of a 
vehicle; movement distance detecting means of detecting the distance of the movement of the vehicle in a unit time; 
^ and space data converting means of converting the space data stored in said space data buffer using the direction 
and the distance of the movement of the vehicle. 

[0116] A fifth applk^tion configuration of an image generating apparatus in accordance with the present invention 
is characterized by comprising: camera c^n-ection indicating means of indicating a camera c;alibratlon to the driver 
when detecting a situation requiring the calibration of the camera; and conrecdon history recording means of recording 

50 the date, time, and mnning distance when the camera calibration was performed. 

[01 17] An image generating method of the present Invention is characterized by comprising: a space reconstructing 
step of reconstructing the space data in whk:h each pixel cx)nstituting the image input from a camera is corresponded 
to a point of the three-dimensional space depending on the camera parameters indk^ating the camera characteristtes; 
and a point-of-view converting step of synthesizing an image viewed from an arbitrary point of view by referencing to 

55 said space data. 

[0118] A first application configuration of an image generating method in accordanc^e with the present invention is 
characterized by comprising a calibrating step for obtaining the camera parameters indk^ating said camera character- 
istics by an input or a cateulation and for (X)rreGting said camera parameters depending on the temperature when 
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necessary. 

. [01 19] A second application configuration of an image generating method in accordance with the present invention 
is characterized by comprising a characteristic point extracting step of extracting a piurality of characterise points 
necessary for the calculation of said camera parameters in said calibrating means. 
5 [01^] A third application configuration of an image generating method in accordance with the present invention is 
characterized by comprising a characteristic point generating step of generating, within the field of view of the camera, 
a plurality of points the three-dimensional coordinates of which are identifiable. 

[01 21 ] A fourth application configuration of an image generating method in accordance with the present invention is 
chara^erlzed by comprising: a movement dire^ion detecting step of detecting the direction of the movement of a 
10 vehicle; a movement distance detecting step of detecting the distance of the movement of the vehicle in a unit time; 
and a space data converting step of converting said space data using the direction of the movement of the vehicle 
detected in said movement direction detecting step and the distance of the movement of the vehicle detected in said 
movement distance detecting means. 

[0122] A fifth application configuration of an Image generating method in accordance with the present invention is 
15 characterized by comprising: a camera conrection indicating step of indicating a camera calibration to the driver when 
detecting a situation requiring the calibration of the camera; and a correction history recording step of recording the 
date, time, and running distance when the camera calibration was performed. 

[0123] An image generating apparatus of the present invention (an example of Claim 1) perfomns the procedure 
consisting of the following three steps, thereby composing the respective fields of view of plural cameras Installed, 
20 thereby synthesizing a single image. 

1 . The space reconstructing means calculates the comespondence relation between each pixel constituting the 
image obtained from a camera emd a point in the three-dimensional coordinate system, thereby generating the 
spac:e data. Said c»lcxilation is perfomned for alt the pixels of the images obtained by the cameras. 
25 2. The point-of-view converting means indicates a desired point of view. That is, it indicates from what position, in 

what angle, and in what magnification in said three-dimensional coordinate system the image is required to view. 
3. The point-of-view converting means also reproduces the image viewed from said point of view using said spaco 
data, and the displaying means displays it. 

[0124] In an example of an image generating apparatus in accordance with the combination of the inventions de- 
scribed in Clainr^ 8, 9, and 12, the characteristic point generating means generates a plurality of points the three- 
dimensional cx>onjinates of which are identifiable, in the surroundings of a body of a vehic:le and the tike. The charac- 
teristic point extracting means extracts those ctiaracteristic points, thereby automatically obtaining the c:amera param- 
eters indicating the characteristics of each camera. 

[0125] An image generating apparatus of the present invention (an example of Claim 14) comprises a temperature 
sensor and a temperature correction table, thereby con-ecting the slight lens distortion which occurs due to the rise or 
fail of the temperature, thereby optimizing the lens. 

[0126] An image generating apparatus of the present invention (an example of Claim 17) provides a method of 
viewing the image of a part which is a dead angle of a camera, as an example of the application of the image generating 
apparatus to a vehicle. That is, the movement direction and the movement distance of the vehicle are detected, and 
the previously acquired image is converted into an image viewed from the present position using a calculation fomnuta 
obtained from the detection result. More specifically, the space data of a place which was seen previously but is unseen 
presently is supplied by converting, through the space data converting means, the space data of the image of said 
placo previously shot which is stored as spaco data in the space data buffer. 

[0127] An image generating apparatus of the present Invention (an example of Claim 1 8), which Is an example of 
the application of the image generating apparatus to a vehicle, detects a situation requiring a camera calibration or the 
correction of camera parameters indicating the camera characteristics, and infomns it to the driver 
[0128] An embodiment of the present invention is described below with reference to the drawings. The present em- 
bodiment Is described for an image generating apparatus in which cameras are provided for monitoring the surround- 
ings of a vehicle and the image acquired by the cameras is displayed on a TV monitor installed near the driver's seat. 
[01 29] Figure 1 is a block diagram showing an example of the basic: configuration of an image generating apparatus 
in accordance with the present invention (Claim 1). 

[0130] An image generating apparatus In accordance with the present invention, as an example of the basic config- 
uration, comprises: a plurality of c;ameras 101 attached for grasping the situation of a monitoring objective, region; a 
camera parameter table 103 for storing the camera parameters indicating the characteristics of said cameras; space 
reconstructing means 1 04 of generating space data in which the image input from said camera is mapped into a space 
model of three-dimensional space depending on said camera parameters; a space data buffer 105 of temporarily 
storing the spaco data generated by said space reconstructing means 104; point-of-view converting means 106 of 
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synthesizing an image viewed from an arbitrary point of view by referencing to the space data; and displaying means 
1 07 of displaying the image converted by said point-of-vlew converting means 106. 

[0131] Figure 2 is a block diagram showing an example of the configuration of an image generating apparatus In 
accordance with the combination of the present inventions descnbed in Claims 8, 9 and 12. 

5 [0132] in the example of Rgure 2. the image generating apparatus shown in Figure 1 further comprises: calibrating 
means 102 of obtaining the camera parameters indcating said camera characteristics, such as a camera attachment 
position, a camera attachment angle, a camera lens distortion conrection value, and a camera lens focal length, by an 
input or a calculation; characteristic point generating means 1 09 of generating, within the field of view of said camera, 
a plurality of points the three-dimensional coordinates of which are identifiable; and characteristic point extrac^ng 

w means 108 of extracting those characteristic points; thereby pennitting to easily obtain the camera parameters indi- 
cating the characteristics of each camera. 

[01 33] The detail of a method for obtaining the camera parameters by the function of the characteristic point gener- 
ating means 1 09 and the characteristic point extracting means 1 08 is described later 

[0134] Figure 3 is a block diagram showing an example of ttie configuration of an image generating apparatus in 
15 accordance with the present invention (an example of Claim 14). In the example of Figure 3, ttie image generating 
apparatus shown in Figure 1 further comprises a temperature sensor 110 and a temperature correction table 111, 
thereby permitting to connect the slight lens distortion whk>h occurs due to the rise or fall of the temperature, thereby 
optimizing the lens. Tbe detail of a method of comecting the lens distortion due to the temperature by the calibrating 
means 102 is described later. 

20 [0135] Figure 4 is a block diagram showing an example of the configuration of an image generating apparatus in 
accordance with the present invention (an example of Claim 1 7). Figure 4 is an example of the configuration of an 
image generating apparatus as an example of the application to a vehicle. The image generating apparatus shown in 
Figure 1 furtiier comprises: movement direction detecting means 112 of detecting the direction of the movement of a 
vehicle; movement distance detecting means 113 of detecting the distance of the movement of the vehble in a unit 

25 time; and space data converting means 114 of converting the space data stored in said space data buffer using the 
direction and the distance of the movement of the vehicle. 

[0136] Using these means, the space data of a place whteh was seen previously but is unseen presently can be 
supplied by converting, through the space data converting means 114 constituting the present invention, tiie space 
data of the previously seen image which Is stored as space data in the space data buffer 105. The detail of a method 

30 of supplying is described later. 

[0137] Figure 5 is a block diagram showing an example of the configuration of an image generating apparatus in 
accordance with the present invention (an example of Claim 18) as an example of the application to a vehicle. 
[0138] In the example of the apparatus of Figure 5, the image generating apparatus shown in Figure 1 further com- 
prises: camera correction indicating means 1 1 6 of indicating a camera calibration to the driver when detecting a situation 

35 requiring the calibration of the camera; and correction history recording means 115 of recording the date, time, and 
running distance when the camera calibration was perfomned. Using these means, a situation requiring a camera 
calibration or the correction of camera parameters indicating the camera characteristks is detected and infomied to 
the driver. 

[0139] Figure 6 is a block diagram showing an image generating apparatus as the integration of Rgures 1 to 5. This 
40 is an example of a combination of the image generating apparatuses of Figures 1 to 5, and pemiits the utilizing of the 
integrated effects obtained by respective configurations. At the last of the present invention, an example of the operation 
of an image generating apparatus in accordance with the present invention will be described with reference to the 
configuration example of this Figure 6. 

[0140] The detail of each component constituting the present invention is described below. 
45 [0141] The camera is a TV camera for shooting an image of a space to be monitored, such as the surroundings of 
a vehicle. The camera used has preferably a large angle of field in order to obtain a large field of view. Figure 7 is a 
conceptual diagram showing an example of the attaching of a camera onto a vehicle. 

[0142] Figure 7 shows an example in which six cameras are installed on the roof of a vehicle so as to overvlewing 

the surroundings from the vehcle. When the attachment position onto the vehicle is a boundary portion between the 
50 roof and the side surface or between the roof and the rear surface as shown In the example of Figure 7, a wide field 
of view is obtained and a smaller number of cameras will do. 

[01 43] The calibrating means 1 02 in accordance with the present invention performs camera calibration. The camera 
calibratbn is the process of detemnlning and correcting the camera parameters Indteating said camera characteristics, 
such as a camera attachment position and a camera attachment angle in the real thrae-dimensional worid, a camera 
55 lens distortion correction value, and a camera lens focal length, for each camera installed in the real three-dimensional 
worid, 

[0144] The camera parameter table 103 in accordance with the present invention is a table for storing the camera 
parameters obtained by the calibrating means 102 (the detail of the process is described later). 
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[0145] As a preparation for the detailed description of the camera parameter table 1 03, a three-dimensional space 
coordinate system is defined. The above-mentioned Figure 7 is a conceptual diagram showing the situation of the 
Installation of cameras onto a vehicle, and showing a three-dimensional space coordinate system based on the vehicle. 
In the example of Figure 7, a three-dimensional space coordinate system as an example of the three^imensional 
5 space coordinate system is defined by: 

• X axis which is the straight line on the road surface just below the rear surface of the vehicle, parallel to the rear 
surface, 

• Y axis which is the axis extending vertfcally fonm the road surface In the center of the rear surface of the vehicle, and 
10 • Z axis which is a straight line on the road surface, perpendicular to the rear surface, through the center of the rear 

surface of the vehicle. 

The orientation of the camera In this coordinate system Is expressed using a and p defined as follows. 

15 • a is the angle relative to the Y-Z plane, and 

• p is the angle relative to the X-Z plane. 

The three-dimensional space coordinate system, the world coordinate system, and simply, the three-dimensional space 
hereafter refer to the three-dimensional space coordinate system defined above, unless otherwise mentioned. 
20 [01 46] Figure 9 is a diagram showing the data stored in the camera parameter table 1 03, in the form of a table. The 
contents described in Figure 9 is as follows, in the sequence starting from the left column of the table. As shown below, 
the itenf^ in the second to the ninth columns In this table show an example of the camera parameters. 

First column: number of a car-carried camera of Figure 7 
25 Second column: X coordinate of a camera position In the three-dimensional space coordinate system 

Third column: Y coordinate of a camera position in the three-dimensional space coordinate system 

Fourth column: Z coordinate of a camera position in the three-dimensional space coordinate system 

Fifth column: angle a relative to the Y-Z plane, among the camera orientation 

Sixth column: angle P relative to the X-Z plane, among the camera orientation 
30 Seventh column: focal length of a camera in the three-dimensional space coordinate system 

Eighth column: distortion coefficient k 1 in the radial direction of a lens 

Ninth column: distortion coefficient k 2 in the radial direction of a lens 

[0147] For example, the parameters of the camera 1 of Figure 7 are described in the second line of the camera 
3s parameter table 1 03 of Figure 9. The contents indicates that the camera 1 is located at a coordinates (x1 , y1 . 0), that 
the orientation is 45 degrees relath/e to the Y-Z plane and-30 degrees relative to the X-Z plane, that the focal length 
is f 1 , and that the lens distortion coefficients k1 and k2 are both zero. 

[0148] Simllariy, the parameters of a virtual camera are described in the eighth line of the camera parameter table 
1 03 of Figure 9. The contents Indicates that the virtual camera Is located at a coordinates (0, y1 , 0), that the orientation 
4a Is 0 degree relative to the Y-Z plane and -20 degrees relative to the X-Z plane, that the focal length is f, and that the 
lens distortion coefficients k1 and k2 are both zero. 

[0149] The virtual camera is a concept introduced in the present invention. Although a prior art image generating 
apparatus can display only an Image obtained by an actually Installed camera, an image generating apparatus In 
accordance with the present invention penmits to obtain an image viewed from the virtual camera located arbitrarily, 
45 by calculation using the space reconstructing means 1 04 and the polnt-of-view converting means 106 described later 
in detail. A method of this calculation is also described later in detail. 

[01 50] The calibrating means 1 02 in accordance with ttie present invention perfonDS camera calibration. That is the 
determination of said camera parameters. The methods of the determination include amethod in which all the data is 
manually input directly using an Inputting device such as a key board and a mouse, a method in which some of the 

so calibration data is obtained by calculation, and the like. 

[01 51 ] The camera parameters are the parameters indicating said camera characteristics, such as a camera attach- 
ment position and a camera attachment angle in a reference coordinate system, a camera lens distortion connection 
value, and a camera lens focal length. When a large number of sets of the points where the correspondence relation 
between the characteristic point in the Image shot by a camera and the position of said characteristic point within said 

55 reference coordinate system is established are known, said parameters can be obtained approximately by calculation. 
In other words, in order to obtain the camera parameters by calculation, a plurality of sets of the points where the 
correspondence relation between the point in the image shot by the camera and the position of that point within the 
three-dimensional space coordinate system is established are necessary. The minimum number of the required sets 
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depends on the calculation method used. For example, a method of obtaining the camera parameters used In the 
example of Rgure 9 by calculation is disclosed in the literature 'Matsuyama, Kuno, and Imiya, "Computer vision: a 
technical critique and a perspective to the future,' The New Technology Communications, pp. 37-53. June 1998.' A 
large number of another technology of obtaining the camera parameters are also disclosed in said literature, and hence 
5 the description of the art is omitted herein. 

[0152] However, regardless of the method used to calculate the camera parameters, it is a problem how to find a 
set of said con-espondence relation. The present invention discloses a method for generating a set of said correspond- 
ence relation, thereby obtaining some of the camera parameters by calculation using the set. This method is described 
later. 

10 [0153] The performing of the lens distortion correction on the camera input image using said tens distortion coeffi- 
cients needs a lot of calculation, and hence is not suitable for a real-time process. 

[0154] Thus, assuming that the variation in the lens distortion does not occur unless the temperature charige is 
vigorous, the correspondence relation between the coordinate values of each pixel before the distortion correction and 
those after the distortion correction is calculated previously. Then, said calculation result is retained in a memory in a 
IS data format such as a table and a matrix, thereby correcting the distortion using it. Such a method is effective as a fast 
correction process. 

[0155] When the change in the lens distortion due to the temperature and the like is previously known, the data is 
retained in a format such as the temperature correction table 111 in accordance with the present invention. The cali- 
brating means 1 02 perf onus the correction by referencing to the data in said table when a change in the lens distortion 
20 occurs due to the rise or fall of the temperature. 

[01 56] Figure 1 0 is a diagram showing an example of the temperature correction table 1 1 1 , in the form of a table. As 
shown in Figure 1 0, the temperature correction table 1 1 1 stores the data of the amount of the change in said camera 
parameters con^esponding to the temperature change. The present example shows the case to store the data of the 
magnitudes of the changes in 

25 

• lens focal length (the second column in the table), 

• lens distortion coefficient k1 (the third column in the table), and 

• lens distortion coefficient k2 (the fourth column in the table) 

30 due to the temperature. More specific contents shown in the table is as follows. 
O When the temperature is 0 degree or lower (the second tine in the table) 

• Add df1 to the present lens focal length. 

35 • Add k1 1 to the present lens distortion coefficient 

• Add k21 to the present lens distortion coefficient. 

O When the temperature is 40 degrees or higher (the third line in the table) 

40 • Add df2 to the present lens focal length. 

• Add k1 2 to the present lens distortion coefficient. 

• Add k22 to the present lens distortion coefficient. 

[0157] The calibrating means 1 02 in accordance with the present invention (an example of Claim 14) successively 
45 observes the temperature value of the temperature sensor 1 1 0 of each camera, and renews the contents of the camera 
parameter table 103 when necessary. 

[0158] Figure 22 is a flow chart showing the procedure of the process of renewing a camera parameter table 103 
depending on the temperature, in calibrating means 1 02. The detail thereof is described below with reference to Figure 
22. 

50 [01 59] Here, in the present example, a temperature sensor 1 1 0 is correspondingly attached to each camera as shown 
in Figure 3. The temperature sensed by the temperature sensor 1 1 0 Is assumed to be approximately equal to the lens 
temperature of the camera. 

1 . (1301) Select one of the temperature sensors 110 not yet checked, and acquire the temperature value. 
55 2. (1 302) Check whether said temperature value requires camera parameter correction or not. 

In the example of Figure 22, the temperature requiring the correction is 0 degree or lower and 40 degrees or 
higher. 

3. (1303) When the correction is necessary, acquire camera parameter correction values from the temperature 
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corra^on table 111 , and write In the result of the renewing of the camera parameters of the camera associated 
with the temperature sensor 110 in question, into the camera parameter table 103. (1304) When the correction is 
unnecessary, write In the result of the restoring all of the lens focal length and the distortion coefficients k1 and k2 
to the Initial setting values, into the camera parameter table 103. 
5 4. (1305) When the abovennentioned processes 1 to 3 are finished for all the twnperature sensors 110, tenninate 

the renewal process of the camera parameter table 1 03. When there is a temperature sensor 1 1 0 not yet checked, 
perform the above-mentioned processes 1 to 3 for this tenr^erature sensor 110. 

[0160] Figure 11 shows an example of a camera parameter table 103 rewritten using the temperature connection 
10 table 111 of the example of Rgure 1 0. In the example of Figure 11 , the cameras 1 and 2 alone are in a temperature of 
40 degrees or higher t>ecause of being inBdiated by direct sun light and the like. The others keep a temperature from 
0 degree less than 40 degrees. As shown in the camera parameter table 1 03 of Rgure 1 1 , the camera parameters of 
the cameras 1 and 2 are changed by the temperature correction process for a temperature of 40 degrees or higher as 
follows: 

15 

• The lens focal lengths are Increased by df 1 . 

• The lens distortk>n coefficients k1 are increased by k 1 2. 

• The lens distortbn coeffictents k22 are increased by k22. 

20 Since the above-mentioned virtual camera can be assumed to have an ideal lens without any change in focal length 
and lens distortion due to the temperature, it Is not an objective of the connection process. 

[01 61 ] In the present example, it is assumed that the lenses of all the installed cameras have the same temperature 
correction characteristics. However, In practtee, there may be a case that the lenses having the different characteristics 
are installed. In that case, preparing an independent table for each camera, the proper table can be used for each 
2S camera of the objective of the temperature correction . 

[0162] Although the material of the lenses of said cameras may be made of plastics other than glass, the plastk^ 
normally undergoes a vigorous distortion due to the temperature change. However, the problem can be treated by the 
above-mentioned connection. 

[0163] The space reconstructing means 104 in accordance with the present invention generates the space data in 
30 vihfch each pixel constituting the image input from a camera is corresponded to a point of the three-dimensional space 
depending on the camera parameters calculated by the calibrating means 1 02. That is, the space reconstmcting means 
104 calculates where in the three-dimensional space each object included in the Image shot by a camera exists, and 
stores the space data as the calculation result Into the space data buffer 105. 

[0164] Here, in the generating of the space data, it is unnecessary to use all the pixels constituting the image from 
35 said camera. For example, in case that the input Image contains a region above the horizon, the pixels included in the 
region above the horizon are unnecessary to map onto the road surface. The pixels constituting the body of vehicle 
are also unnecessary to map. Further, in case of an input image of a high resolution, the process may be speeded up 
by mapping every several pixels into the space data. 

[01 65] Generally, the position of each pixel constituting the Image shot by a camera is expressed by the coordinates 
40 on the U-V plane Including the CX^D imaging plane. Thus, in order to establish the con^espondence of each pixel 
constituting the input Image Into a point of the worid coordinate system, a calculation fonnula is necessary whfch 
establishes the correspondence of a point of the U-V plane Including the image shot by the camera into a point of the 
worid coordinate system. 

[0166] Figure 8 is a conceptual diagram representing an example of the correspondence relation between a point 
45 of the U-V coordinate system being set on a plane (plane of view hereafter) including the Image shot by a camera and 
a point of the three-dimensional space coordinate system. In accordance with the example of Rgure 8, the con^spond- 
ence relation is established by ttie following procedure. 

1 . Set a coordinate system in which the plane of view is Z = f (focal length of a camera) and the Z axis passes 
so through the center of the camera image on the plane. This is called the plane-of-view coordinate system (the origin 

is Oe). 

2. Let the coordinates of a point Pe of Figure 8 in the plane-of-view coordinate system be Pe (Xe, Ye, Ze), and let 
the coordinates of the projected point of ttiat point onto the plane of view (this point con^esponds to a single pixel 
of the image shot by the camera) be Pv (u, v). Then, using the focal length f of the camera, the relation between 

55 Pe and Pv is expressed as following equations (1 ) and (2). 
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U = ^ Xg Equation 1 



V = =- Equation 2 

The above-fnentioned two equations can determine the coordinates, in the plane-of-view coordinate systm, 
of each pixel of the image projected onto the plane of view. 

3. Obtain the positional relation and the orientational relation between the plane-of-vtew coordinate system and 
the world coordinate system. IHere, assume that, in space, the plane-of-view coordinate system is in the following 
relation with the world coordinate system centered on. 

Let the vector from the origin Oe of the plane-of-view coordinate system to the origin Ow of the worid coordinate 
system be (tx, ty. tz). Then, the positional shift between the two coordinate systems vanishes by the parallel 
displacement by (tx, ty, tz). 

Set the orientational relation between the plane-of-vlew coordinate system and the worid coordinate system 
so as to be the same relation between the coordinate system (corresponding to the worid coordinate system) 
based on the vehicle of the example of Figure 7 and the car-canied camera (corresponding to the plane-of- 
view coordinate system) . 

Then, in the plane-of-view coordinate system, 

"the angle relative to the Y-Z plane of the worid coordinate system is a" and 
"the angle relative to the X-Z plane of the worid coordinate system is p." 

Here, the rotation of the camera lens around the optical axis Is assumed to be neglected. In case that a point is 
represented by the coordinates Pw (Xw, Yw, Zw) in the worid coordinate system and by the coordinates Pe (Xe, 
Ye, Ze) in the plane-of-view coordinate system, the relation of Equation (3) holds among Pe (Xe, Ye, Ze), Pw (Xw, 
Yw, Zw), (be, ty, tz), a, and p. 
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Now, the correspondence between a pixel Pv (u, v) on the plane of view and the coordinates Pw (Xw, Yw, Zw) in 
the worid coordinate system has been established by Equations (1), (2), and (3). 

In the above-mentioned three equations, since the unknown parameters are the six of "to, ty, tz, a, p, and f 
the above-mentioned unknown parameters can be detemnined by having at least two sets of the points where the 
correspondence relation between the pixel Pv (u, v) on the plane of view and the coordinates (Xw, Yw, Zw) in 
the worid coordinate system is known. 

However, the measurement of the coordinates of each set the correspondence relation in which is known 
contains a positional shift in most cases, whteh causes an error in the cateulation result. 

Therefore, a method normally used is to solve the simultaneous equations constructed by the measurement 
of a large number of the sets, by the metiiod of least squares. Here, since the position and the orientation of the 
camera agree with the plane^f-view coordinate system, the present process 3. is unnecessary to perfomi when 
the camera parameters are previously known. 

4. Using Equations (1 ), (2), and (3) in whk^h the values of tx, ty, tz, a, p, and f are given, establish the con^espondence 
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of each point Pw (Xw, Yw, Zw) of the world coordinate system into a point Pv (u, v) on the plane of view. 

[01 67] By performing the above-mentioned processes 1 to 4 on aii of the cameras, the con-espondence of the inde- 
pendent images from ali of the cameras into the points in the same three-dimensional space can be established. 

5 [01 68] The space data in accordance with the present invention is the data in which the correspondence is established 
by such calculation formulas. Figure 12 shows an example of the description of the space data buffer 105 for storing 
the space data, in the fomr) of a table. The space data buffer 105 stores the correspondence data between a point of 
a camera image and a point within the space. In the example of Rgure 12, a piece of the space data is described in 
each line except for the first line. Each line, which Is the Inf omnation constituting each space data, includes the following 

10 contents. 

First column: number for identifying the point from the others in the world coordinate system (A is used here for 
the sake of the simplicity of description) 

Second column: X coordinate of the point in the world coordinate system 

IS Third column: Y coordinate of the point in the worid coordinate system 
Fourth column: Z coordinate of the point in the worid coordinate system 
Fifth column: flag for identifying the camera which shot the image including said point 
Sixth column: U coordinate of the point in the plane-of-view coordinate system including said image 
Seventh column: V coordinate of the point In the plane-of-view coordinate system including said image 

20 Eighth column: R color component (quantized, for example, in 0-255 gradations) of the point In the plane-of-view 

coordinate system including said image 

Ninth column: G color component (quantized, for example, in 0-255 gradations) of the point in the plane-of-view 
coordinate system including said image 

Tenth column B color component (quantized, for example, in 0-255 gradations) of the point in the plane-of-view 
25 coordinate system Including said image 

Eleventh column: time when the space data is acquired 

The contents of the description of the space data Is explained below with an example. Rgures 13 to 1 5 are used 
for the assistance of the explanation. 
30 [0169] Figures 13 to 15 are diagranns showing the correspondence relation between a characteristic point on the 
road surface as a plane In the worid coordinate system and a characteristic point on the Innage shot by a camera 
mounted on the vehicle. Figure 13 is a conceptual diagram of the positional relation between characteristic points A, 
B, C, D, and E on the road surface and the vehicle, viewed from above. Figure 14 is a conceptual diagram showing 
an image of the road surface including said char^eristic points A, B, and C. shot by the car-carried camera 1 of Figure 
35 15. Figure 15 is a conceptual diagram showing an image of the road surface including said characteristic points C, D, 
E, shot by the car-carried camera 2 of Figure 13. In the space data buffer 105 of the example of Figure 12, the five 
characteristic points A, B, C, D, and E of Rgures 13 to 15 are described as an example of the space data. 
[01 70] First, consider the characteristic point A in Rgures 1 3 and 1 4. Assuming that the point A in the worid coordinate 
system of Rgure 13 is con-esponded to the point A in the plane of view coordinate system of Figure 14 by the come- 
do spondence process of said space reconstructing means 1 04, the third line in the table of Figure 12 is the example of 
the space data corresponding to the characteristic point A of Figures 13 and 14. 

[0171] This example indicates that the point A in the worid coordinate system has coordinates (X3, 0, Z2). When it 
is shot by the camera 1 , the coordinates of the point A on the camera image are (U1 , VI ). The colors in the order of 
R, G, B are (80, 80, 80), and the time of the generation of the data is t1 . 

^ [01 72] When a point in the worid coordinate system is shot by a plurality of cameras, each of the Independent data 
is stored In the space data buffer 1 05. An example of this case is the point C of Figures 1 3 to 1 5. As shown in Figures 
14 and 15. the point C is observed by the two cameras of Figure 13, the camera 1 and the camera 2. 
[0173] In the space data generated on the basis of the olaservation result by the camera 1 , which is the space data 
in the seventh line of Figure 12, the point C in the worid coordinate system has coordinates (0, 0, Z2). When the point 

50 c Is shot by the camera 1 , the coordinates of the point C on the camera image are (U3, V3). The colors in the order 
of R, G, B are (140, 140, 140), and the time of the generation of the data is t1 . 

[01 74] On the other hand, in the space data generated on the basis of the observation result by the camera 2, which 
is the space data in the eighth line of F^ure 12, the point C in the worid coordinate system has coordinates (0, 0, Z2). 
When the point C is shot by the camera 2, the coordinates of the point C on the camera image are (U4, V4). The colors 
55 in the order of R, G, B are (1 50, 1 50, 1 50), and the time of the generation of the data is ti. 

[0175] As such, the information of the con^espondence of each pixel of the Image shot by each camera into a point 

of the worid coordinate system is stored into the space data buffer 105 in the fomn of the space data. 

[0176] The point-of-view converting means 108 in accordance with the present invention generates an image shot 
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by a camera placed at an arbitrary point of view by referring to the space data generated by the space reconstruc^'ng 
means 104. The outline of the method is to perform the reversed process of the process carried out by the space 
reconstmcting means 104. That is, it is to obtain the transfomiation of projecting a point Pw (Xw, Yw, Zw) generated 
in the worid coordinate syst^ by the space reconstructing means 104 onto the image plane Pv (u, v) shot by the 
camera placed at the arbitrary point of view. 

[0177] The calculation formula for this transformation can be expressed by Equation (4) which con^esponds to the 
inverse transformation of Equations (1). (2), and (3) described above in detail. 
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[01 78] That is, by inputting Pw (Xw, Yw, Zw), the Pv (u, v) is calculated using said three equations. In this calculation, 
arbitrary and desired values can be specified for the camera parameters of the camera "tx, ty, tz, a, p, and f." This 
indicates that the camera can be placed at a desired point of view and at a desired angle. In that case, there occurs 
a problem how to represent the color of each pixel on the screen projecting the image viewed from the camera placed 
at said arbitrary point of view. 

[0179] However, the present invention (corresponds to Claims 5, 6, and 7) discloses the methods of displaying the 
color for three cases. The contents of the methods is described below for each of the three cases. 



(1) The case that a point P in the three-dimensional space viewed from said point of view is corresponded to the 
image shot by a single camera: in this case, the color of the point P viewed from the set point of view is detemnined 
using the color of the point P viewed by said camera. The simplest method is to replace it with the same color 

30 [However, the color may be obtained by calculation depending on the relation between the positrons and the direc- 

tions of the set point of view and of the camera observing the point P. 

(2) The case that a point P in the three-dimensional space viewed from said point of view is corresponded to the 
images shot by a plurality of cameras: in this case, the color of the point P viewed from the set point of view is 
determined by the color of the result of a certain calculation using the color of the point P viewed by said plurality 

35 of cameras. Some methods can be considered as the methods of said calculation, which are, for example, 

• to mix all the colors in the same ratio, 

• to acquire the color having the highest, the lowest, or the middle lightness, and 

• to acquire the color having the highest, the lowest, or the middle saturation. 

40 

(3) The case that a point P in the three-dimensional space viewed from said point of view is not corresponded to 
an image shot by any cameras: in this case, the color of the point P viewed from the set point of view is determined 
by the calculation to obtain the color interpolated with the colors of the points In the surroundings of the point P, 
by replacing it with a color, such as blade, identifiable as a part where no objects exist, and by the like. 

45 

[0180] The major feature of the present invention is the capability of arbitrarily reproducing an image viewed from 
an virtual camera which is not installed on the vehicle, by the point-of-view converting means 106. 
[01 81 ] For example. Figures 1 6 to 1 9 are conceptual diagrams showing an example in which an image viewed from 
a virtual camera placed at a proper point of view is synthesized with the images of the characteristic points on the road 

so surface within the worid coordinate system shot by the cameras installed on the vehicle. Figure 16 is a conceptual 
diagram of the positional relation between characteristic points A, B, C on the road surface and the vehicle, viewed 
from above. Figure 17 is a conceptual diagram showing an image of the road surface including said characteristic 
points A and B, shot by the car-canled camera 1 of Rgure 1 6. Figure 1 8 is a conceptual diagram showing an image 
of the road surface including said characteristic points B and C, shot by the car-camed camera 2 of Figure 16. Figure 

55 19 is a conceptual diagram showing the situation of the synthesizing of an image viewed from a virtual camera, by 
polnt>of-view converting means 1 06 In accordance with the present invention using the irnages shot by the car-carried 
cameras 1 and 2 of Figure 16. 

[Om] Figure 20(a) is a conceptual diagram showing the case that a camera is installed above the approximate 



17 



EP1 115250A1 



10 



IS 



20 



25 



center of a vehicle downwardly, as an example of the place of installation of said virtual cannera. In case of installing 
the virtual camera as this example, the image shot by the virtual camera shows the surroundings of the vehicle. Since 
the innages constituting said synthesized image are shot by the car-cam'ed cameras, the roof of the body of the vehicle 
is not Included in any camera image when the surroundings of the car-camed cameras Is shot in the configuration 
shown in Rgure 7. 

[0163] However, concerning an object of which the infonnation of the existence position, the shape, and the color 
are known such as said roof of the body of the vehide, a less strange image can be synthesized by previously storing 
such information In the system and by using the information when necessary. 

[0184] Figure 20(b) is a perspec^e diagram showing the case that a camera is installed obliquely above the ap- 
proximate front side of a vehicle, directing the camera to the vehicle. As such, the position of the virtual camera is not 
restricted to that from just above, and the vehicle can be viewed obliquely. Figure 20(c) is a synthesized diagram of 
the image generated by u^ng the Figure 20(b), which represents the feeling as viewed obliquely from above. 
[0185] The space data converting means 114 in accordance with the present invention is means necessary when 
the image generating apparatus described in Claim 1 of the present invention is applied to a vehicle. 
[0186] Generally, a car-carried camera is nomnalty installed on top of a body of a vehide in order to obtain a better 
field of view in many cases. However, in case that the body of the vehide has a shape fomiing an outwardly convex 
surface, for example, the road surface in the near surroundings of the vehide is almost always within a dead angle in 
the image shot by such a camera located on top of the body of the vehide. 

[0187] A simple method to resolve the problem is to install a camera also in a lower part of the body of the vehicle. 
However, the addition of the camera needs an extra cost. Said space data converting rheans 11 4 of the present invention 
resoh^es the problem without the addition of a camera in a lower part of the body of the vehide or the like. The solution 
depends on an assumption that the vehtele moves. 

[01 88] Figure 23 is a flow chart showing the procedure of the process in space data converting means 1 1 4. Rgure 
24 is a conceptual diagram used for the assistance of the description of space data converting means 114. Rgure 24 
shows the relation between the vehicle positions and the vehicle directions at the starting time (t1 hereafter) and at 
the ending time (t2 hereafter) of a predetennlned time duration during the vehicle movement. The procedure of syn- 
thesizing the image of the part in a dead angle from the camera is described belowwith reference to Figures 23 and 24. 
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1 . (1 401 ) Detect the movement distance of the vehble In a predetennlned time duration. In the present example, 
the movement distance is defined by the straight distance between the vehide positions at time tl and at time t2. 
That is the distance between 01 and 02 in Figure 24. 

For the convenience of description, the movement distance from 01 to 02 is represented by a vector (t^, 0, 
tz) as shown In Rgure 24. The method used for the detection of the movement distance is, for example, a method 
by counting the number of revolutions of the wheel. 

2. (1402) Detect the direction of the movement of the vehide in said predetermined time duration. In the present 
example, the movement direction is defined by the amount of the change which is the difference of the direction 
of the vehicle at time t2 from the direction of the vehicle at time tl . For the convenience of description, the amount 
of the change in the direction is represented by the angle e between the Z1 axis and the 72 axis as shown in 
Figure 24. The method used for the detection of the movement direction is, for example, to measure the rotation 
angle of the steering wheel or the like. 

3. (1403) Formulate Equation (5) for converting the space data acquired at tl into the space data at t2 using the 
movement distance and the movement direction of the vehicle during the time duration from time tl to time 12. In 
Equation (5), it is assumed that there is perfectly no change In the vertk»l component in the vehide movement 
during the time duration from time tl to time 12, that is, the road surface is flat. 



Equa-bloa 5 



r \ 

X2 

y2 

Z2 



cos^ 0 s'm 



0 1 0 

S\T\9 0 COS* 



0 

tz 

V yy 



In Equation (5), the x1 , y1 , and z1 represent the coordinates of a point in the XI -Y1 -Z1 wortd coordinate system 
(the origin 01 based on the body of the vehicle at time tl , and the x2, y2, and z2 represent the coordinates of said 
point in the X2-Y2-Z2 worid coordinate system (the origin 02) based on the body of the vehteie at time t2. That is, 
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the x2, y2. and z2 are the result of the calculation of substituting x1 , y1 . and z1 into the right hand side of Equation 
(5). 

4. (1 404) Using Equation (5), convert the space data generated at time tl into the space data at time 12. Since the 
generated space data does not need the information which camera is used for viewing, the data in the fifth to the 
5 seventh columns in the table of Figure 12 may be left blank. That is. among the space data at time t1 . only the 

first to the fourth columns in the table of Figure 12 are rewritten, and the data in the eighth to the eleventh columns 
are used intact. 

[0189] A problem is that the overflow in the limited space data buffer 105 will occur some time when the past space 
10 data is successively added to the present space data as descnbed at>ove. Since each space data in the space data 

buffer 105 in accordance with the present invention has the infomnatlon of the time of the generation of the data, the 

data prior to a predetermined past time back from the present may be deleted to resolve the problem. 

[01 90] The characteristic point generating means 1 09 in accordance with the present Inventton generates, within the 

field of view of a camera, a plurality of points the three-dimensional coordinates of which are identifiable. The charac- 
15 teristic point extracting means 108 extracts said generated characteristk: points. Figure 21 is a conceptual diagram 

showing an embodiment of the characteristic point generating means 1 09, characteristic points, and the characteristk: 

point extracting means 108. 

[01 91 ] Figure 21 (a) is an embodiment in which a pattern light irradiating apparatus as characteristc point generating 
means 1 09 is attached on top of a side of the body of a vehicle. In the present example, rectangle pattems are in^diated 
20 in the form of a grid on the road surface as a plane in the world coordinate system, in the surroundings of the vehble. 
[0192] Figure 21(b) is an example of the situation that a plurality of the pattern light irradiating apparatuses are 
attached on top of the body of a vehk^ie thereby inradiating the pattern light onto the road surface, viewed from top of 
the vehicle. 

[0193] Figure 21 (c) Is an example of the situation that the rectangle pattern light inadiated onto the road surface by 
25 such a method is shot by a camera. Certain points, such as the comers and the center of a rectangle generated by 

the pattem light irradiation, representing the characteristics can be used as the characteristic points. 

[0194] In Rgure 21(c), the PI-1 to Pt-8 are the examples of a characteristic point. It is possible to set so that the 

coordinates of said characteristk: points are known in the worid coordinate system. Further, the coordinate positbns 

of these characteristic points are also known in the plane-of-view coordinate system, and hence the correspondence 
30 relation between the worid coordinate system and the plane-of-view coordinate system is established. Accordingly, 

using the above-mentioned Equations (1), (2), and (3), the calibrating means 1 02 can cateulatesthe camera parameters 

tx,ty,tz,a,p,andf. 

[01 95] The correction Indicating means in accordance with the present Invention Is means necessary when the Image 
generating apparatus described in Claim 1 of the present invention is applied to a vehble. The correction indicating 
35 means detects a situation requiring the calibration of the camera, and indicates the camera calibration to the driver 
when the camera calibration is necessary. The correction history recording means 115 in accordance with the present 
invention records the date, time, and running distance when the camera calibration was performed, as tiie data nec- 
essary to detect the situation requiring the camera calibration. 

[0196] Figure 25 is a flow chart showing the procedure of the process of checking the record of correction history 
40 thereby Issuing a correction indk^ation when nec:essary. 

1 . (1 601 ) Calculate the elapsed time from the time of the previous camera calibration to the present 

2. (1602) When said elapsed time is greater than a predetermined time set beforehand, (1605) the camera cot- 
rection indk:ating means 116 indicates the performing of camera c^alibratksn to the driver, and then terminates the 

45 process. When the driver perfomns the camera calibration according to said indication, the recx>rd of conrection 

history is renewed . When said elapsed time is smaller than said predetermined time, proc^eed to the next prcx^ess 3. 

3. (1603) Calculate the total running distance from the time of the previous camera calibration to the present. 

4. (1 604) When said running distance is greater than a predetennined distance set beforehand, (1 605) the camera 
correction indicating means 116 indicates the perfonning of camera calibration to the driver, and then temninates 

50 the process. When the driver perfonns the camera calibration according to said lndk:ation, the record of correction 

history is renewed. When said running distance is smaller than said predetennined distance, the camera calibration 
indbation is not issued to the driver, and the process is temninated. 

[0197] An embodiment of each means constituting an image generating apparatus in accx>rdance with the present 
55 invention has been described above. The flow of the overall process of the image generating apparatus in accx)rdance 
with the present invention is described below. 

[01 98] Figure 26 is a flow chart showing the flow of the overall process of the image generating apparatus in accord- 
anc:e with the present invention in the case of tiie applcation to a vehkte. The configuratbn of Figure 6 is assumed 
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as the example of the configuration of the image generating apparatus. 

1 . (1 701 ) In order that the present apparatus can operate normally, pert onn the camera calibration initially if nec- 
essary, and renew the record of correction history. In the camera calibration, the camera parameters may be input 

5 manually, or calculated by the calibrating means 102 using the result of the extraction of characteristic points by 

the characteristic point extracting means 108 after the characteristic point generating means 109 in accordance 
with the present invention generates the characteristic points in the surroundings of the body of the vehicle. 

2. (1702) Observe the temperature value of the temperature sensor 110 of each camera successively while re- 
newing the contents of the camera parameter table 1 03 when necessary. 

10 3. (1 703) Check the record of correction history and issue a correction indication when necessary. When the cor- 

rection is performed, renew the record of correction history. 

4. (1704) When the space data is stored in the space data buffer 105, the space data converting means 114 
converts said space data depending on the mov^ent distance and the movement direction of the vehicle. When 
the space data buffer 1 05 is blank, the present process is omitted. 
is 5. (1 705) The car-carried cameras shot the images of the sun-oundings of the vehicle. 

6. (1706) The space reconstructing means 104 generates the space data in which each pixel constituting the 
images shot in 5, is corresponded to a point of the worid coordinate system. When the space data of whk^h coor- 
dinates in the worid coordinate system coincides are present among the space data converted In 3, by the space 
data converting means 1 1 4, said converted space data is disposed. That is, only the latest data shot by the camera 

20 is retained among the space data of a specific point in the worid coordinate system. The data prior to that time 

and the data of which time has been elapsed in a certain extent are deleted. 

7. (1 707) Referring to the space data generated by the space reconstructing means 1 04 in 6., compose an image 
shot by a camera installed at a desired point of view. In this case, the point of view is preferably fixed at a place 
suitable for the assistance of the driving. An example of this is a camera position above the body of the vehicle 

25 from where the sun-oundings of the vehicle can be overviewed, as the example of Figure 20. 

8. (1708) Display the image synthesized in the process of 7. 

9. Repeat the above-mentioned processes 2. to 8. until the necessity is lost. For example, the above-mentioned 
processes 2. to 8. are repeated during the driver is trying to park the vehicle in a paridng space, and the present 
processes are temninated on completion of paridng. 

30 

[0199] If the three-dimensional position of an object, such as an obstacle In the sun-oundings of the vehkHe, can be 
accurately measured in the worid coordinate system, the space model can be generated correspondingly to the object. 
However, it is nonrnally impossible in practice. 

[0200] That is, in a system, such as an Image generating apparatus, requiring the simplicity, it is difficult to accurately 
35 obtain the three-dimensional positions and shapes of all the objects. Further, from the aspect of synthesizing an image 
as the assistance of driving, it is not definitely necessary to reproduce an Image viewed from an arbitrary virtual point 
of view in an absolute accuracy. Thus, even somewhat low quality scarcely causes a large problem as tong as the 
image is easy to underatand for the driver. 

[0201 ] Accordingly, the present invention (an example of Claim 3) discloses a method of establishing the correspond- 
40 ence between each pixel constituting the input image and a point of the three-dimensional space, wherein the three- 
dimensional infomnation of the object is lost, but an image can be synthesized at a high speed, and the quality of the 
synthesized image is kept somewhat. 

[0202] The method is to project the image obtained by a camera of which the attachment position and the attachment 
angle of which are previously known onto the road surface as an example of a plane constituting a part of the three- 

45 dimensional space. In other words, assuming that each object included in the image is sticking onto the X-Z plane of 
the three-dimensional space coordinate system (called the worid coordinate system sometime hereafter) and that there 
is no object having a component in the Y axis direction, the image on the plane of view is projected onto the road surface. 
[0203] I n this case, the only alteration to the contents of the above-rrtentioned description of the embodiments of the 
present apparatus is that Equation (3) used by the space reconstructing means 104 is replaces by Equation (6) and 

so that Equation (4) used by the point-of-view converting means 106 is replaces by Equation (7). 
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[0204] Although the description of the present embodiment has been made for an apparatus for monitoring the sur- 
roundings of a vehicle using the cameras attached on the vehicle, the present art of synthesizing a image viewed from 
an arbitrary point of view using a limited numt)er of cameras is not restricted to the car-carried cameras. 
[0205] For example, by installing a large number of watching cameras In a shop and by using those camera Images, 
an image viewed from just above can be synthesized. As such, a wide range of application can be expected. 
[0206] Another possible method is to install the above-mentioned calibrating means in a management center in a 
remote place and use it via conrtmunicating means. 

[0207] In a method and an apparatus of the present invention, it is of course possible to implement a part of the 
function in such a remote place. 

[0208] Further, the embodiments of the present invention includes the case that the data of the calibration result is 
carried using a floppy disk, a DVD, and the like and then used. 

[0209] The buffer for storing the mapped space data is not definitely necessary when the space data is processed 
directly. 

[0210] The virtual point of view is nonnally not specified manually by the manager, the driver, or the like but one of 
the polnt-of-view positions providing an image useful for the assistance of monitoring or driving is selected and the 
image viewed therefrom is displayed. As such, the operation of moving the virtual point of view can be eliminated, 
thereby pemittting a further reduction in the woricing load to the user 

[0211] As such, in accordance with the present inventton, an Image viewed from an arbitrary point of view can be 
synthesized using the images from a limited numt>er of cameras. 

[0212] An image generating apparatus in accordance with the present Invention (an example of the combination of 
the inventions described in Claims 8, 9, and 12) permits to easily obtain the camera parameters indicating the char- 
acteristics of each camera. Even in case that the position of a camera is somewhat shifted during the mnning on a 
very rough road, the present invention permits easy correction of the camera position. 

[0213] An image generating apparatus in accordance with the present invention (an example of Claim 14) pemiits 
to correct the slight lens distortion whk>h occurs due to the rise or fall of the temperature, thereby optimizing the lens 
by setting a temperature sensor and a temperature comection table. For example, for a lens slightly expanded due to 
the temperature rise, the lens distortion parameters can be changed using the connection values for optimally controlling 
the lens distortion coefficients whk:h change depending on the expansion, by obtaining them from a temperature cor- 
rection table. By such a method, the present invention permits to obtain a distortion-free image at any temperature. 
[0214] An image generating apparatus in accordance with the present invention (an example of Claim 1 7) provides 
a method to watch an image of a part in a dead angle from a camera. For example, in case that the cameras are 
attached on top of the body of the vehicle and that the shape of the body of the vehicle below the camera attachment 
position Is an outwardly convex surface, it Is physbally impossible to view the image just below the cameras. However, 
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in the present invention, the image acquired previously can be converted to an innage viewed from the present position 
depending on the movement direction and the movement distance of the vehicle. 

[021 5] An in^ge generating apparatus In accordance with the present invention (an example of Claim 1 8) pennrtting 
to detect a situation requiring the correction of the camera parameters or the camera calibration which shows the 
5 characteristic of the camera, and to indicate that to the driver. As such, there is an effect to prevent the driver from 
forgetting to carried out the correction of the camera parameters for a long time. 

[0216] An embodiment of another present invention Is described below. Rgures 27 and 28 show an example of the 
present Invention (Qaims 35 and 37). 

[0217] As shown in Figure 27(a), an innage generating apparatus in accordance with the present invention, as an 
10 example of the basic configuration, has the configuration comprising: a plurality of cameras 101 A attached for grasping 
the situation of the surroundings of a vehicle; a camera parameter table 102A for storing the camera parameters 
Indicating the characteristics of said cameras; space model generating means 103A for generating a space model In 
a coordinate system based on the vehicle; mapping means 1 04A for mapping the Image input from said camera, into 
said space model; point>of-vlew converting means 105A for setting a point of view and synthesizing a single image 
IS viewed from said point of view using the data generated by said mapping means 104A; and displaying means 106A 
for displaying the image converted by said point-of-view converting means 1 05A. 

[0218] Figure 27(b) is a block diagram showing an example of the configuration of an Image generating apparatus 
in accordance with the present invention. In the example of Rgure 27(b), the configuration is the image generating 
apparatus shown in Rgure 27(a) further added obstacle detecting means 1 08A for measuring at least the distance to 
20 an obstacle existing in the surroundings of the vehicle, as the situation of the surroundings of the vehicle. 
[021 9] The detail of each component constituting the present invention Is described below. 
[0220] The camera is a TV camera for shooting an Image of a space to be monitored, such as the surroundings of 
a vehicle. The detail thereof is described in the abovenfnentioned Rgure 7. 

[0221 ] The camera parameter table 1 02A in accordance with the present invention is a table for storing the camera 

25 parameters (similar to the above-mentioned camera parameter table 1 03). 

[0222] The data stored in the camera parameter table 1 02A is the same as that of the above-mentioned Figure 9. 
[0223] In the present invention, assuming that a virtual camera is installed at a desired point of view set by the point- 
of-vlew converting means 1 05A, the image viewed from the virtual camera can be obtained by calculation. The method 
of the calculation is described later. 

30 [0224] The performing of the lens distortion correction on the camera input image using said lens distortion coeffi- 
cients needs a lot of calculation, and hence is not suitable for a real-time process. 

[0225] Thus, assuming that the variation in the lens distortion does not occur in the extent of causing a problem in 
the Image composition, the con-espondence relation between the coordinate values of each pixel before the distortion 
correction and after the distortion correction is calculated previously. Then, said calculation result Is retained in a mem- 
35 ory in a data format such as a table and a matrix, tiiereby correcting the distortion using it. Such a method Is effective 
In a fast correction process. 

[0226] The space model generating means 103A generates a space model, for example, In the coordinate system 
with the reference on the vehicle. The space model is a model consisting of one or more plane(s), one or more curved 
surface(s), or one or more ptane(s) and one ore more curved surface(s) where the Image from a camera Is mapped 
40 into the three-dimensional space coordinate system by the mapping means 1 04A described later. Rgures 28(a) to 28 
(c) are conceptual diagrams showing a space model in a bird's eye view. Figures 28(a) and 28(d) show examples of 
a space model composed of planes alone. Figure 28(b) shows an example composed of a curved surface alone, and 
Figure 28(c) shows an example composed of a plane and a curved surface. 

[0227] The space model of Rgure 28(a) shows a space model composed of five planes described below. 

45 

Plane 1 : the plane of the road surface (in contact with the wheels) 
Plane 2: a plane vertical to the road surface (Plane 1 ) on the front side 

Plane 3: a plane vertical to the road surface (Plane 1) on the left side relative to the direction of the traveling of 
the vehicle 

50 Plane 4: a plane vertical to the road surface (Plane 1 ) on the rear side 

Plane 5: a plane vertical to the road surface (Plane 1) on the right side relative to the direction of the traveling of 
the vehicle 

[0228] In the present space model, Planes 2 to 5 is set without any gap, and hence the image shot by a car-carried 
55 camera is mapped any of Planes 1 to 5. With regard to Planes 2 to 5, the necessary distance from the vehicle and the 
necessary height can be determined depending on the angle of field and the place of installation of the car-carried 
camera. 

[0229] In the model of Rgure 28(b), the space model uses a curved surface of a bowl shape. The vehicle Is located 
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on the part corresponding to the bottom of the bowl in the space model of the bowl shape. The image shot by the car* 
carried camera is mapped onto the inner surface of the bowl. The models of a bowl shape Include a sphere, a parabola 
of revolution, a catenary of revolution, and the like. Anyway, a space model expressed by few equations permitting a 
fast calculation of the mapping. 
5 [0230] The model of Figure 28(c) shows a space model composed of the combination of the plane and the curved 
surface descrik)ed below. 

Plane: the plane of the road surface (in contact with the wheels) 

Curved surface: a wall of cylindrical or elliptic-cylindrical shape placed on said plane so as to sunround the vehicle 

10 

[0231 ] In the present space model, the shape of the curved surface and the necessary distance from the vehicle can 
be detemnined depending on the angle of field and the place of Installation of the car-carried camera. 
[0232] Such a space model having a wall in the surroundings of the vehicle so as to surround the vehicle has the 
following effect. That is, when the camera image is mapped onto the road surface assuming that all the objects in the 

15 image are located on the road surface, there has been a problem that an object having a height above the road surface 
is substantially deformed. In contrast, in the space model introduced by the present invention, the vehicle is surrounded 
by a plane or a curved surface vertical or substantially vertical to the road surface. By setting these surfaces not too 
far from the vehicle, the object having a height component is mapped onto these surfaces, permitting to reduce the 
distortion. Further, because of the small distortion In the mapping, the discrepancy in the junction part of the images 

20 by two cameras is expected to t>e small. 

[0233] The mapping means 104A In accordance with the present invention maps each pixel constituting the input 
image from the car-canied camera into a space model generated by the space model generating means 103 in ac- 
cordance with the present invention. That is, it perspectively projects each image shot by the car-canied camera into 
the space model. 

25 [0234] Figure 29 is a diagram used for the assistance of the description of the mapping of a car-carried camera 
image onto a surface constituting the space model by converting the coordinates of a point in the U-V coordinate 
system being set in the plane (called the plane of view hereafter) including the image into the coordinates of a point 
of the worid coordinate system. 

[0235] Hefore the description of the mapping, the method for converting the plane-of-view coordinates to the worid 
30 coordinates is described betow. This conversion is perfomied in the following procedure. 

1 . Set a coordinate system in which the plane of view is Z = f (lens focal length) and the Z axis passes through the 

center of the camera image on the plane. 

This is called the plane-of-view coordinate system (let the origin be Oe). 
35 2. Let the coordinates of a point Pv (u, v) (this point connesponds to a pixel of the camera shot Image) of Figure 29 

in the plane-of-view coordinate system be Re (Xe, Ye, Ze). Then, using the focal length f of the camera, the relation 

between Pe and is expressed by Equations 1 and 2 (where Ze = f in this case). 

The above-mentioned two equations can determine the coordinates, in the plane-of-view coordinate system, 

of each pixel of the image projected onto the plane of view. 
40 3. Obtain the calculation formula for establishing the correspondence between the plane-of-view coordinate system 

and the worid coordinate system. Here, assume that the plane-of-view coordinate system is In the following relation 

with the reference to the worid coordinate system. 

Let the vector from the origin Oe of the plane-of-view coordinate system to the origin Ow of the worid coordinate 
^ system be (tx, ty, tz). Then, the positional shift between the two coordinate systems vanishes by the parallel dis- 

placement by (tx, ty, tz). 

When the camera position is known in the worid coordinate system, the (tx, ty, tz) is easily obtained by inverting 
the sign of the coordinates of the position . 

so , Let the rotation matrix for aligning the orientation of each axis of the plane-of-view coordinate system with that of 
the worid coordinate system be 
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This rotation matrix can be easily obtained by obtaining the rotation matrix around each of the X, Y, and Z axes based 
on the orientation parameters in the camera parameter table 102A and by making the composition of them. 
[0236] Here, when a point is represented by Pw (Xw. Yw, Zw) In the world coordinate system and by Pe (Xe, Ye, Ze) 
In the plane-of-view coordinate system, the relation of Equation (8) holds among Pe (Xe, Ye, Ze), Pw (Xw, Yw, Zw), 
(tx, ty, tz), and the above-mentioned rotation matrix. Using these Equations (1), (2), and (8), a pixel Pv (u, v) on the 
plane of view can be converted to the coordinates Pw (Xw, Yw, Zw) In the world coordinate system. 
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[0237] The position of each pixel constituting the image shot by the car-camed camera is generally represented by 
the coordinates on a plane including the image plane. In order to map each pixel constituting the input image into said 
space model, the process In the following procedure is performed for all the pixels of the input image. 
[0238] Figure 30 is a flow chart showing the procedure of the mapping process. The contents of the process of the 
mapping is described below with reference to Figure 30. 



1 . Convert said pixel represented by the coordinates of the U-V coordinate system being set in the plane of view 
of the camera Image Into the coordinates of the world coordinate system. The conversion can be performed, for 
example, by Equations (1), (2), and (8) shown just previously. 
55 2. Obtain the intersecting point Ps (Xs, Ys, zs) between the half straight line starting at the origin Oe of the camera 

point of view and passing through the coordinates Pw (= Pe) of said pixel and the plane constituting said space 
model. 

3. Map the color of said pixel into the intersecting point Ps. When the color from the image of another camera has 
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been already mapped in to the point Ps, the color of the point Ps can be detennined, for example, by the following 
method. 

• to mix, in the same ratio, the color already mapped and the color to be mapped. 

• to use the color having the higher, the lower, or the middle tightness between the color already mapped and 
the color to be mapped, and 

• to use the color having the higher, the lower, or the middle saturation between the color already mapped and 
the color to be mapped. 

[0239] By perfonning the above-mentioned processes 1 to 3 for all the cameras, the images from all of the inde- 
pendent cameras can be mapped into the space model in the same three-dimensional space. 
[0240] The point-of -view converting means 1 05 in accordance with the present invention composes an image shot 
by a camera installed at an arbitrary point of view from the result of the mapping of the car-carried camera image into 
the space data by the mapping means 1 04. The outline of the method is to perf omn the reversed process of the process 
canied out by the mapping means 1 04. 

[0241] That is, to be obtained is the transfomnation of projecting a point Ps (Xs, Ys, Zs) constituting the mapped 
image onto the image plane Pv (u, v) shot by a camera installed at an arbitrary point of view. 
[0242] Thus, the calculation fomnula for this conversion can be expressed by Equations (1 ), (2), and (9) (correspond- 
ing to the inverse transformation of Equation (8)) descnbed above in detail. 

[0243] That is, by inputting Ps (Xs, Ys, Zs), the Pv (u, v) is calculated using said three equations. In this calculation, 
arbitrary and desired values can be specified by camera parameter. This indicates that the camera can be placed at 
a desired point of view and at a desired angle. 

[0244] Meanwhile, during the composition of the image viewed from a camera placed at said arbitrary point of view 
by the point-of-view converting means 1 05 A, a situation can occur that no color is being mapped in a point of the space 
model corresponding to a pixel of said synthesized image. In that case, it can be replaced with a color, such as black, 
identifiable as a part without an object. 

[0245] Although the examples of Figures 28(a) to 28(c) show a space models In which the vehicle is completely 
surrounded by a surface, the case is rare that all the surroundings of the vehicle is surrounded by ok>stacles. The 
situation Is more normal that at most several vehicles are present in the surroundings of the vehicle in question. Con- 
sidering such situations, instead of introducing a space model completely surrounding the surroundings of the vehicle 
in question, a plane like a screen can be generated only depending on the necessity, that is, only in front of obstacles 
when the situation of the distribution of obstacles in the surroundings of the vehicle in question is known. 
[0246] Figure 28 (d) Is a conceptual diagram showing an example of the space model introducing the screen planes, 
in a bird's eye view. Figure 28 (d) Is a conceptual diagram showing a space model having three screen planes each 
on the road suriace and on the rear left side and the rear right side of the vehicle on the road surface. Although the 
screen planes can be previously set at predetermined places as described above, a screen plane may be set only 
when an obstacle having a height component above the road surface is found In the surroundings of the vehble. In 
that case, it is necessary to determine the position and the direction of the screen plane to be set. As an example of 
this, the method of setting a screen plane depending on the result of the detection of the obstacle is described below. 
[0247] In order to perfomi the setting of such a screen plane, the ot)stacte detecting means 1 08 in accordance with 
the present invention measures at least the distance to an obstacle existing in the surroundings of the vehicle, as the 
situation of the surroundings of the vehble, using a distance sensor 1 07A. 

[0248] Various apparatuses can be used as the distance sensor 1 07A. However, the car-carried distance sensor can 
utilize laser light, ultrasonk: wave, a stereo optical system, a camera focusing system (the distance from the cannera 
to said object is calculated from the focal length when focused on the objective), and the like. In case that the distance 
sensor 107A uses the laser light, the ultrasonk: wave, and the like, a large number of the apparatuses are desirably 
attached in the periphery of the vehble. In case of using the stereo optical system or the camera focusing system, it 
may be installed in addition to the car-carried camera, however, use of a part of the car-carried camera pemnrts to 
reduce the cost. 

[0249] An example of the method of generating a space model included a screen plane depending on the process 
result of the obstacle detecting means 1 08A is described below. Figure 31 is a conceptual diagram showing the method 
of setting a screen plane in the three-dimensional space depending on the distance between the vehicle and an obstacle 
existing In the surroundings of the vehicle measured by an obstacle sensor. In the present example, it Is assumed that 
each car-canied camera Is equipped with an obstacle detecting sensor mentioned above in addition thereto. The 
obstacle sensor is directing to the same direction as the line of sight of the camera, and measures the distance to an 
obstacle existing In the direction. The parameters used in Figure 31 is as follows. 

(pxO, pyO, pzO) : coordinates of the distance sensor 107 1 on the road surface plane 
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(dx, dy, d2) : direction vector indicating the orientation of the sensing of the distance sensor 107 1 

dc : distance on the road surface plane between the distance sensor 1 07 1 and the obstacle 

(px1 . py1 . pz1 ) : the point shifted from (pxO, pyO, pzO) in the direction of (dx, dy. dz) by the distance dc 

s [0250] The procedure of generating the space nrK)dei is described below with reference to Figure 31 . 

1 . Measure the distance dc to the obstacle and check whether the distance is smaller than a predetermined dis- 
tance. When the distance is larger than said predetermined distance, nothing is performed. 

2. When within the predetenmined distance, obtain a screen plane by the following procedure. 

2-1. Obtain (pxO, pyO, pzO) from the previously known coordinate values of the distance sensor 107A. 

2-2. Obtain (px1 , py 1 . pzl ) from both the previously known direction (dx, dy, dz) of the distance sensor 1 07A 

and the distance dc to the obstacle. 

2-3. Set the plane (Equation (15)) having the normal vector (dx, dy, dz) and passing through the point (pxl. 
15 pyi , pz1 ) as the screen plane. 

dx (x-pxl ) -I- dy(y-py1 ) + dz (z-pzl ) = 0 Equation 1 5 

20 3. Remove said screen plane when the necessity is lost. 

4. Terminate the space model generating process when the above-mentioned processes 1 to 3 have been com- 
pleted for all the distance sensors 1 07. When there is a distance sensor 1 07A not yet checked, perform the above- 
mentioned processes 1 to 3 on the distance sensor 1 07A. 

25 [0251] Figure 31 shows the situation that the images input from the cameras 1 and 2 are mapped onto the planes 1 
and 2, respectively, as the screen planes composing the space model. An important factor determining the quality of 
the synthesized Image Is the value of the width of the screen plane being set In the mapping. Since the most frequent 
obstacle is another vehkde, the width of the screen plane can be detennined so that the 2/3 or nrtore of said another 
vehicle is mapped onto the screen plane when the distance between the vehicle in question and the another vehicle 

30 becomes the distance to generate a screen plane, for example. 

[0252] With regard to the condition to determine whether a screen plane is to be generated or not, the distance from 
the vehicle in question to an obstacle can be set to be 50 cm to 1 m, from experience. With regard to the condition to 
remove the screen plane, the following conditions can be used solely or in combinatbn. 

35 • The distance to the obstacle has become larger than a predetermined distance. 
• The driver has initialized the image generating apparatus by any method. 

[0253] An embodiment of each means constituting an innage generating apparatus In accordance with the present 
invention has been described above. The flow of the overall process of the image generating apparatus In accordance 
40 with the present invention is described below. Figure 32 is a flow chart showing the flow of the overall process of the 
Image generating apparatus in accordance with the present invention. The configuration of Figure 27(b) Is assumed 
as. the example of the configuration of the Image generating apparatus. 

1 . (901) In order for the present apparatus to operate normally, correct camera parameters for each car-carried 
45 camera are previously input into the camera parameter table 1 02A. 

2. (902) The car-carried camera shots an image of the surroundings of the vehicle. 

3. (903) The obstacle detecting means 1 08A measures the distance to an obstacle existing in the surroundings of 
the vehble using the distance sensor 107A. 

4. (904) The space model generating means 103A generates an space model. 

50 5. (905) The mapping means 1 04A maps the image by the car-carried camera into the space rTK>del. 

6. (906) An image viewed from a point of view set by the drh^er is synthesized by referencing to the image mapped 
in the space model. 

7. (907) The Image synthesized by the process 6. Is displayed. 

8. (908) The above-mentioned processes 2. to 7. are repeated until the necessity is lost. For example, the above- 
55 mentioned processes 2. to 7. are repeated during the driver is trying to park the vehkde in a parking space, and 

the present processes are terminated on completion of parking. 

[0254] As such, in accordance with the present invention, an image viewed from an arbitrary point of view is synthe- 
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sized using the images from a Itmited number of cameras. In the composition, an space model other than a conven- 
tionally used space model having a road surface plane alone is introduced. Using this space model, even an object 
having a height is mapped into the space with reduced distortion. Accordingly, when an object having a height is shot 
into the images of two cameras, the discrepancy In the overiap of the images generated by the mapping of said object 

5 Into the space model is substantially Improved In comparison with the plane model, wheretiy the quality of the image 
synthesized by the point-of-vlew conversion is improved. Thus, the situation of the surroundings is more easily recog- 
nizable for the driver with said synthesized image, thereby permitting a suitable driving operation. 
[0255] The above-mentioned description has been made in the case that the surroundings of the vehicle Is detected 
from the vehicle and that the coordinate system with the reference on the vehicle is used. However, the above-nrrcn- 

10 tioned prcx^ess can be perfonned, for example, even in the case that a sensor installed in a parking field and the like 
detects the situation of the parking field and Infomns, to the vehicle, t>oth the situation of the surroundings in the coor- 
dinate system with the reference in the parking field and the positional relation of the vehicles in the parking field. 
[0256] An embodiment of another present Invention is described bek)w. The present embodiment Is described for a 
monitoring apparatus in which cameras are provided for monitoring the surroundings of a vehicle and the image ac- 

is quired by the cameras is displayed on a TV monitor installed near the driver's seat. 

[0257] Figure 33(a) is a block diagram showing an example of the basic configuration of a monitoring apparatus in 
accordance with the present invention (an example of Claim 39). 

[0258] An apparatus for monitoring the surroundings of a vehk^le in accordance with the present emt>odiment has, 
as the basic configuration, the configuration comprising: a plurality of cameras 1 01 B attached for grasping the situation 

20 of the sunoundings of a vehble; a camera parameter table 102B for storing the camera parameters indicating the 
charac^eristtes of said cameras; road-surface characteristics detecting means 1038 for detecting, as the situation of 
the surroundings of the vehicle, the characteristns on the road surface, such as a white line and an arrow, a character, 
and a crosswalk drawn on the road surface; space model generating means 104B for setting a coordinate system, for 
example, with the reference on the vehicle and generating, in the coordinate system, a space model depending on the 

25 process result of said road-surface characteristics detecting means 1 03B; mapping means 1 05B for mapping the Image 
input from said camera, into said space model; point-of-view converting means 106B for setting a point of view and 
synthesizing a single Image viewed from said point of view using tiie data generated by said mapping means 105B; 
and displaying means 107B for displaying the image converted by said polnt-of-view converting means 106B. 
[0259] Figure 33(b) Is characterized, In comparison witii the monitoring apparatus shown In Figure 33(a), by further 

30 composing movement direction detecting means 109B for detecting the movement direction of the vehicle and the 
movement distance detecting means 108B for detecting ttie movement distance of the vehicle in a unit time and by 
using the process results thereof thereby to calculate the present position of said characteristics on the road surface 
thereby to conrect successively said space model depending on the calculated present position of the vehicle. 
[0260] Figure 33(c) has the configuration further comprising, in comparison with the monitoring apparatus shown in 

35 Rgure 33(b), characteristics oonrecting means 1 1 0B for displaying the process result of the road-surface characteristics 
detecting means 103B on the displaying means 107B and connecting said process result. As a result, when the char- 
acteristics on the road surface shifts during the process, the shift can be con-ected. 
[0261] The detail of each conrtponent constituting the present embodiment is described below. 
[0262] The camera is a TV camera for shooting an image of a space to be monitored, such as the surroundings of 

40 a vehicle. The camera used has preferably a large angle of field in order to obtain a large field of view. An example of 
the attachment of the cameras are as described in Figure 7. 

[0263] The camera parameter table 1 02B is a table for storing the camera parameters. The contents thereof is as 
described above. 

[0264] The data stored in the camera parameter table 1 02B is expressed in the form of a table as shown in Figure 9. 
45 [0265] Simllariy, the parameters of a virtual camera are described in the eighth line of the camera parameter table 
1 02 of Figure 9. The contents indicates that the virtual camera Is located at a coordinates (0, y1 , 0), that the orientation 
Is 0 degree relative to the Y-Z plane and - 20 degrees relative to the X-Z plane, that the focal length is f , and that the 
lens distortion coeffk;ients k1 and ic2 are both zero. 

[0266] Assuming that a virtual camera is installed at a desired point of view set by the polnt-of-^iew converting means 
so 106B, the image generating apparatus in accordance with the present invention penults to obtain an image viewed 
from a virtual camera by calculation. A method of this calculation is described later 

[0267] The perfonning of the lens distortion correction on the camera input image using said lens distortion coeffi- 
cients needs a bt of calculation, and hence is not suitable for a real-time process. 

[0268] Thug, assuming that the variation In the lens distortion does not occur in the extent of causing a problem in 
55 the Image composition , the conrespondence relation between the coordinate values of each pixel before the distortion 
correction and after the distortion con-ection is calculated previously. Then, said calculation result is retained in a mem- 
ory in a data format such as a table and a matrix, thereby correcting the distortion using It. Such a method is effective 
in a fast correction process. 
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[0269] The space model generating means 1 046 sets a coordinate system with the reference on the vehicle and 
generates, in the coordinate system, a space model depending on the process result of the road-surface characteristics 
detecting means 103B. 

[0270] Figures 34(a) and 34(b) are conceptual diagrams showing a space model in accordance with the present 
5 invention. 

[0271 ] The present figure shows the situation that an end point of a white line drawn in a parking field for Indicating 
the parking space or a comer formed by the Intersecting lines is detected as the characteristics on the road surface 
and that a space model using the five planes shown in the figure is generated with the reference on those points. Rgure 
34(a) is a conceptual diagram showing the space model in a bird's eye view, and Figure 34(b) is a diagram in a per- 
10 spective projec^on view from above the vehicle downwardiy. In the example of Figure 34(b), characteristic points 1 to 
4 are shown as the examples of the characteristbs on the road surface. The characteristics on the road surface or the 
characteristks indicates these four characteristic points hereafter. 

[0272] Here, when any of the four is to be mentioned. It Is specified with a numeral as "characteristic point 1 In the 
present embodmient, the five planes are defined as follows (here, right and left are defined when directing to backward 
15 of the vehicle). 

Plane 1 : the plane of the road surface (in contact with the wheels) 

Plane 2: the plane vertical of Plane 1 and the left edge of whk:h contacts to Plane 2 

Plane 3: the plane vertical to Plane 1 and aligning with the line element connecting the characteristk: point 1 and 
20 the characteristic point 2 

Plane 4: the plane vertical to Plane 1 and aligning with the line element connecting the characteristk: point 2 and 
the characteristb point 3 

Plane 5: the plane vertical to Plane 1 and aligning with the line element connecting the characteristic point 3 and 
the characteristic point 4 
25 Plane 6: the plane vertical of Plane 1 and the right edge of whteh contacts to Plane 5 

[0273] I n order for the space model generating means 1 04B to generate these planes ( except for Plane 1 which is 
identical to the road surface) In the three-dimensional space, said characteristic points need to be detected. The road- 
surface characteristics detecting means 1 03B extracts such characteristic points on the road surface. 
30 [0274] Figures 35(a) to 35(d) are diagrams for desc^bing an example of the process of extracting a characteristic 
point by roaci-suriace characteristics detecting means 1 038. Figure 36 is a flow chart showing the flow of the process 
of extracting a characteristic point. The prcx^edure of the process by the road-suriace characteristk^s detecting means 
103B Is described below with reference to these figures. 

[0275] Process 701 : Shot an image including a white line indicating the parking space using any one of the car- 
35 carried cameras. Figure 35(a) shows the shot image. 

[0276] By the following two proc:esses, estimate the position of a characteristk: point. 

[0277] Process 702: Binarize the image shot by the car-carried c^amera with an appropriate threshold, scan it In the 
horizontal and the vertical directions, and obtain a histogram, in each of the horizontal and the vertical directions, of 
the number of the pixels corresponding to the white line In the scanning line. Estimate the position of a characteristk: 
40 point from the result of the histograms. Figure 35(b) shows an example of the process, and shows the situation that 

• Y coordinate of the ctiaracteristic point is estimated from the histogram in the scan in the vertical direction 

• X coordinate of the characteristic point is estimated from the histogram in the scan in the horizontal direction 
Process 703: Perfonrt the ecige extracting process on the image shot by the car-c:am'ed camera, then, perform the 

45 straight line extracting process on the result of the edge extraction, and estimate a Intersecting point or a end point 

of the obtained straight lines as a characteristic point. Figure 35(c) shows an example of the process. The edge 
extraction can be performed, for example, by Sobel operator, and the straight line extraction can be perfonned, 
for example, by Hough transformation. 

, 50 [0278] Proc:ess 704: Determine the characteristic point using the estimated value obtain In the above-mentioned 
processes 702 and 703. The method of detemnining the characteristic point is, for example, to adopt the middle point 
of the ctiaracteristic points obtained by the respective processes 702 and 703. Note that the same result is obtained 
regardless of order of performing the processes 702 and 703. 

[0279] Process 705: Obtain the coordinates of the characteristic point obtained from the (»mera image, in the three- 
55 dimensional cx)ordinate system. The planes constituting said space model c:an be obtained from these (K)ordinate8. 
[0280] The mapping means 1 05B maps each pixel constituting the input image from the car-cam'ed camera, into a 
space model generated by the space model generating means 104B depending on the c:amera parameters. That is, 
it perspectively projects each image shot by the c:ar-carried camera into the space model. 
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[0281 ] Figure 29 is a diagrani used as an assistance for the description of the mapping of a car-camed camera image 
onto a surface constituting the space model by converting the coordinates of a point in the U-V coordinate system 
t>eing set in the plane including the image into the coordinates of a point of the world coordinate system. The detail 
thereof has been described above. 

5 [0282] The position of each pixel constituting the image shot by the car-carried camera Is generally expressed by 
the coordinates on the plane including the image plane. The detail thereof has been described above in Rgure 30. 
[0283] The point-of-view converting means 1 06B composes an image shot by a camera Installed at an arbitrary point 
of view from the result of the nnapping of the car-carried camera Image Into the space data by the mapping means 
1 05B The outline of the method has been described above with regard to the above-nnentloned point-of-view converting 

10 means 105A. 

[0284] Meanwhile, during the composition of the image viewed from a camera placed at said arbitrary point of view 
by the point-of-view converting means 1 08B, a situation can occur that no color is being mapped in a point of the space 
model corresponding to a pixel of said synthesized image. In that case, It can be replaced with a color, such as black, 
identifiable as a part without an object. 

IS [0285] The position of the characteristic point detected by the road-surface characteristics detecting means 103B 
changes as the vehicle moves. Since the space model is generated depending on the three-dimensional coordinates 
of the characteristic point, the space model needs to be re-generated as the position of the characteristic point changes 
owing to the movement of the vehicle. That Is, the position of the characteristic point needs to be detemnined, for 
example, by the abovennentioned method or the like, successively during the movement of the vehicle. However, the 

20 process of obtaining the characteristic point from the image needs generally a lot of calculation, thereby causing a 
higher cost. A method for avoiding this is to measure the speed and the direction of the movement of the vehicle 
successively and calculate the coordinates of the characteristic point using the measurement result. 
[0286] In order to perform such a process, an apparatus for monitoring the surroundings of a yehk^e In accordance 
with the present invention (an example of Claim 41 ) composes: movement direction detecting means 1 09B for detecting 

25 the movement direction of the vehicle; and the movement distance detecting means 1 08B for detecting the movement 
distance of the vehicle in a unit time; and uses the process results of the movement direction detecting means 109B 
and the movement distance detecting means 108B thereby to calculate the present position of said characteristics on 
the road surface. 

[0287] Figure 37 is a flow chart showing the procedure of the process of calculating the position of the characteristic 
30 point as the vehicle moves, in an apparatus for monitoring the surroundings of a vehicle in accordance with the present 
Invention (an example of Claim 41). Rgure 38 is a conceptual diagram used for the assistance of the description of 
said process. 

[0288] Figure 38 shows the relation between the vehicle positions and the vehicle directions at the starting time (t1 
hereafter) and at the ending time (t2 hereafter) of a predetennined time duration during the vehksle movement. The 
35 procedure of said process is described below with reference to Figures 37 and 38. 

[0289] Process 901 : Detect the distance of the movement of the vehicle in a predetemnlned time duration. In the 
present embodiment, the movement distance is defined by the straight distance between the vehtole positions at time 
t1 and at time t2. That is the distance between 01 and 02 in Rgure 38. 

[0290] For the convenience of description, the movement distance from 01 to 02 is represented by a vector (t^, 0. 
40 tz ) as shown in Figure 38. The method used for the detection of the movement distance is, for example, to count the 
number of revolutions of the wheel. 

[0291 ] Process 902: Detect the direction of the movement of the vehicle in said predetermined time duration. In the 
present embodiment, the movement direction is defined by the amount of the change whtoh is the difference of the 
direction of the vehicle at time 12 from the direction of the vehicle at time t1 . 
45 [0292] For the convenience of description, the amount of the change in the direction Is represented by the angle 9 
between the Z1 axis and the 72 axis as shown in Figure 38. The method used for the detection of the movement 
direction is, for example, to measure the rotation angle of the driving wheel. 

[0293] Process 903: Formulate above-mentioned Equation (5) for converting the coordinates of the characteristic 
point acquired at t1 into the coordinates of the characteristic point at t2 using the movement distance and the movement 
50 direction of the vehicle during the time duration from time t1 to time t2. 

[0294] In Equation (5), it is assumed that there is perfectiy no change in the vertical component in the vehicle move- 
ment during the time duration from time t1 to time t2, that is, the road surface is flat. 

[0295] In Equation (5), the x1 , y1 , and z1 represent the coordinates of a point in the XI -Y1 -Z1 worid coordinate 
system (the origin 01 based on the vehicle at time t1 , and the x2, y2, and z2 represent the coordinates of said point 
55 in the X2-Y2-Z2 worid coordinate system (the origin 02) based on the body of the vehtele at time t2. That is, the x2, 
y2, and z2 are the result of the calculation of substituting x1 . y1 , and z1 into the right hand side of Equation (5). 
[0296] Process 904: Using Equation (5), convert the coordinates of the characteristte point synthesized at time t1 
Into the coordinates of the characteristic point at time t2. 
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[0297] If the movement dire<^on and the movement distance of the vehide can be measured without an error, exa<^ 
coordinates of the characteristic point can be obtained always by introducing the movement direction detecting means 
109. the movement distance detecting means, and a characteristics position calculating means and by using the cal- 
culation method described just above. However, the measurement of the movement direction and the movement dis- 
5 tance of the vehicle without an error is impossible in practice, and hence the necessity occurs for the con^ec^on of the 
position of the charactertetic point. 

[0298] While displaying the process result of the road-surface characteristics detecting means 1 03B on the displaying 
means 107B, the characteristics correcting means 11 OB indicates the connection of the ctiaracteristics to the crew of 
the vehicle. The crew connects the characteristics position on the road surface according to the indication. 
10 [0299] Figure 39 is a conceptual diagram showing the situation tiiat the characteristics con-ecting process is displayed 
on displaying means 1078. Rgure 40 is a flow chart showing the flow of the process in the characteristics con-ecting 
prcx^ess. The procedure of the process of the characteristics correcting means 1 1 08 is descmbed below with reference 
to these figures. 

[0300] Process 1201: Display the present position of the characteristic point on the displaying means 107B. The 
15 example of Figure 39 shows the situation that each image shot by the car-carried camera and the space model are 
superposed and displayed in a perspectlvely projected image from above downwardly. 

[0301] Process 1202: Specify a characteristic point the position, of which is shifted. In the example of Rgure 39. 
characteristic points 2 and 4 are the shifted points and hence specified. Witii regard to the specifying, when the dis- 
playing apparatus is equipped, for example, with a touch panel, the specification of the place is easily performed by 
20 touching the displaying screen with the finger. 

[0302] Process 1 203: Specify the conrect place of said sh Ifted characteristic point. The operator is previously Inf onmed 
where the correct place is. 

[0303] Process 1 204: When a chara^eristic point to be corrected is still present, r^eat the above-mentioned Proc- 
esses 1201 to 1203. Otherwise, temiinate the correcting process of the characteristic point. 

25 [0304] An embodiment of each means constituting an apparatus for monitoring the sun^oundings of a vehicle in 
acx^ordance with tiie present invention has been desc^bed above. The flow of the overall process of the apparatus for 
monitoring the sunvundings of a vehicle in accordance with tiie present invention is described below. Figure 41 is a 
flow chart showing the flow of the overall proc^ess of the apparatus for monitoring the surroundings of a vehicle in 
accordance with the present invention. The configuration of Figure 33(c) is assumed as the example of the configuration 

30 of the apparatus for monitoring the surroundings of a vehicle. 

[0305] Proc:ess 1301 : In order for the present apparatus to operate nomnally, correct c:amera parameters for each 
c:ar-c:arried camera are previously input into the camera parameter table 1028. 

[0306] Process 1302: Extract a characteristic point on the road surface from the image shot by the car-canied camera. 

[0307] Process 1303: Calculate ttie three-dimentional coordinates of the characteristic point using the within-image 
35 coordinates of the extracted characteristic point and the camera parameters. Here, when the process of the apparatus 

for monitoring the surroundings of a vehicle is in progress, the present position of the coordinates of the characteristic 

point is calculated using the detection result of the movement direction and the movement distance of the vehic:le. 

[0308] Process 1 304: The car-cam'ed camera shots the image of the surroundings of the vehicle. 

[0309] Process 1305: The space model generating means 104B generates a space model for the Image shot in 
40 Procoss 4. 

[0310] Prcxoss 1 306: The mapping means 1 058 maps the image from the car-camed camera into the space model. 
[0311] Process 1307: Compose an image viewed from a point of view specified by the driver, by referencing to the 
image mapped Into the space model. 

[031 2] Process 1 308: Display the image synthesized in Procoss 1 307. 

45 [0313] Process 1309: Check the displayed image whether the position of the characteristic point is shifted or not. 
[0314] Process 1310: When shifted, intennpt for the characteristic point correction and perform the process of the 
characteristic point correction. When the characteristic point correction is unnecessary, return to Process 1303 and 
repeat the processes. For example, the above-mentioned Processes 1302 to 1308 are repeated during the driver is 
trying to parte the vehicle in a paricing space, and the present processes are temninated on conrtpletion of paridng. 

50 [031 5] The situation of the surroundings of a vehicle In accordance with the present Invention includes the condition 
of the pariced vehicles as well as the above-mentioned characteristics on the road surface. In this case, the space 
model is generated con^espondingty to the condition of the partced vehicles. 

[031 6] As such, in accordance with the present Invention, an image viewed from an arii>itrary point of view Is synthe- 
sized using the Images from a limited number of cameras. In the composition, a universal space model generated 
55 using the characteristics on the road surface obtained from the camera image is Introduco Instead of a simple plane 
model. In a method using a plane, since the height component is converted to the depth component in the direction of 
the line of sight, there has been a problem that an object having a height above the road surface is projected onto the 
road surface with large distortion. However, using the space model in accordance with the present invention, even an 
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object having a height is mapped into the space with reduced distortion. Accordingly, when an object having a height 
is shot into the iniages of two cameras, the discrepancy in the overlap of the infiages generated by the mapping of said 
object into the space model is substantially improved in comparison with the plane model, wheretjy the quality of the 
image synthesized by the point-of-view conversion is improved. Thus, the situation of the surroundings is more easily 
5 recognizable for the driver with said synthesized image, thereby permitting a suitable driving operation. Further, said 
space model has a simple configuration theretsy permitting to reduce the cost of the apparatus. 
[0317] Another image generating apparatus of the present invention is described below. 

[0318] An embodiment of the present invention is described below with reference to some figures. Figure 42 is a 
block diagram showing an example of the configuration of an image generating apparatus in accordance with the 

10 present invention (an example of Claim 45). 

[0319] An apparatus for image generating apparatus in accordance with the present embodiment has, as the basic 
configuration, the configuration comprising: a plurality of cameras 1 01 C attached for grasping the situation of the sur- 
roundings of a vehicle; a camera parameter table.1 02C for storing the camera parameters Indicating the characteristics 
of cameras 101C; mapping means 104C for mapping the images input from the cameras 101C, into a space model 

IS 1 03C generated by modeling the situation of the surroundings of ttie vehicle; point^f -view converting means 1 05C for 
synthesizing a single image viewed from a desired virtual point of view using the data generated by the mapping means 
104C; camera parameter correc^ng means 106C for con-e^ng the parameters of the cameras 101C independently 
for each camera; and displaying means 1 07C for displaying the image converted by the point-of-view converting means 
105C. 

20 [0320] The detail of each component constituting the present embodiment is described below 

[0321] The camera 1 01 C is a TV camera for shooting an image of a space to be monitored, such as the surroundings 
of a vehicle. Figure 43(a) is a conceptual diagram showing an example of three cameras installed on a vehicle. When 
the position of the attachment orito the vehicle is a boundary portion between the roof and a side surface or between 
the roof and the rear surface as shown in Figure 43(a) and when the camera used has preferably a large angle of field, 

^5 . a wide field of view is obtained and a smaller number of cameras will do. When a camera directing backward is installed 
in each of the right and left door mirror sections and the image from the camera is displayed on a monitor installed in 
the vehicle, the camera serves as the door mirror. Accordingly, the door mirror can be removed from the vehicle, thereby 
permitting a car design of excellence in aesthetes as well as aerodynamics. 

[0322] The camera parameter table 102C is a table for storing the camera parameters. The detail thereof is as 
30 described above. 

[0323] As a preparation for the detailed description of the camera parameter table 1 02C, a three-dimensional space 
coordinate system based on the vehicle is defined. Figure 44 is a conceptual diagram showing the three-dimensional 
space coordinate system based on the vehicle. In the example of 44, a three-dimensional space coordinate system 
as an example of the three-dimensional space coordinate system is defined by: 

35 

• X axis whk:h is the straight line on the road surface just below the rear surface of the vehk:le, parallel to the rear 
surface, 

• Y axis which Is the axis extending vertically from the road surface in the center of the rear surface of the vehicle, and 

• Z axis whteh is a straight line on the road surface, perpendicular to the rear surface, through the center of the rear 
40 surface of the vehble. 

[0324] The orientation of the camera in this coordinate system is expressed using a, p and y defined as follows. 

• a Is the angle relative to the Y-Z plane, 

45 • p Is the angle relative to the X-Z plane, and 

• Y is the rotation angle around the optical axis of the camera 

after the rotation by the above-mentioned angles a and p. The three-dimensional space coordinate system, the 
worid coordinate system, and simply, the three-dimensional space hereafter refer to the three-dimensional space co- 
so ordinate system defined above, unless otherwise mentioned. 

[0325] Figure 45 is a diagram showing the data stored In the camera parameter table 102C, in the form of a table. 
The contents described in Figure 45 are as follows, in the sequence starting from the left column of the table. As shown 
below, the items In the second to the ninth columns In this table show an example of the camera parameters. 

55 First column: number of a car-camed camera of Figure 44 

Second column: X coordinate of a camera position in the three-dimensional space coordinate system 
Third column: Y coordinate of a camera position in the three-dimensional space coordinate system 
Fourth column: Z coordinate of a camera position in the three-dimensional space coordinate system 
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Fifth column: angle a relative to the Y-Z plane, among the camera orientation 
Sixth column: angle p relative to the X-Z plane, among the camera orientation 
Seventh column : rotation angle y around the optical axis of the camera 
Eighth column: focal length of a camera in the three-dimensional space coordinate system 
5 Ninth column: distortion coefTictent kI In the radial direction of a lens 

Tenth column: distortion coefTictent k2 in the radial direction of a tens 

[0326] For example, the parameters of the camera 1 of Rgure 44 are described in the second line of the camera 
parameter table 1 02C of Figure 45. The contents indicates that the camera 1 1s located at a coordinates (0. y1 , 0), that 

10 the orientation is 0 degree relative to the Y-Z plane, -30 degrees relative to the X-Z plane, and no rotation around the 
optical axis, that the focal length is f 1 , and that the lens distortion coefficients k1 and k2 are both zero. 
[0327] In the present embodiment, the polnt-of-vlew converting means 105C (described later in detail) permits to 
convert an image shot by the car-carried camera into an Image viewed from a desired virtual point of view. SpecTically, 
the image viewed from the virtual point of view is an image which is to be seen if a camera is placed at a desired place 

15 and at a desired orientation. Thus, the point-of-view parameters of the virtual point of view can be expressed by the 
same camera parameters as described above. Here, in the virtual camera lens distortion is unnecessary to consider, 
and hence both of the lens distortion coefficients k1 and k2 can be set to be zero. 

[0328] Figure 46 shows the data stored in a point-of-view parameter table 1 Q2C, in the fomn of a table. The items in 
the second to the ninth columns, from left to right, of the contents descn*be in Figure 48 show an example of the point- 

20 of-view parameters. The contents indicates that the virtual point of view is located at a coordinates (0, 0, z2), that the 
orientation is 0 degree relative to the Y-Z plane, -90 degrees relative to the X-Z plane, and no rotation around the optical 
axis, that the focal length is f2, and that the lens distortion coefficients k1 and k2 are both zero. 
[0329] The rnapping means 1 04C maps each pixel constituting the Input image from the car-carried camera, into a 
space model 103C depending on the camera parameters. That is, it perspectively projects each image shot by the 

25 car-carried camera into the space model 1 03C. 

[0330] Here, the space model 1 03C is a three-dimensional model where the mapping means 1 04C maps the image 
from the camera into the three-dimensional space coordinate system. The model used is, for example, a model con- 
sisting of a plane, a curved surface, or a plane and a curved surface. In the present embodiment, for the simplicity of 
description, a plane model of the road surface Is used as an example of the simplest space model 1 03C. The method 

30 of mapping the image by the car-carried camera into said plane model is described below. 

[0331] In practice, such a simple model causes a problem, for example, that an object having a height component 
is substantially deformed. Thus, a space model consisting of several planes, a screen plane consisting of planes and 
curved surfaces, and the like are more preferable. Further, when an accurate three-dimensional model of the obstacles 
in the surroundings of the vehicle can be obtained in real time, an synthesized image with higher precision is obtainable 

35 using such a three-dimensional model. 

[0332] Before the description of the mapping process by the mapping means 104C, the method for converting the 
plane-of-view coordinates to the worid coordinates is described below. Figure 47 is a diagram used for the assistance 
of the description of the mapping of a point in the U-V coordinate system being set in the plane (plane of view hereafter) 
including the image into a point in the world coordinate system. This conversion is performed in the following procedure. 

40 [0333] Procedure 1 : Set a coordinate system in which the plane of view is Z = f (tens focal length) and the Z axis 
passes through the center of the camera image on the plane. This is called the plane-of-view coordinate system (let 
the origin be Oe). 

[0334] Procedure 2: Let the coordinates of a point Pv (u, v) (this point con^esponds to a pixel of the camera shot 
image) of Figure 7 in the plane-of-view coordinate system be Pe (Xe, Ye, Ze). Then, using the focal length f of the 
45 camera, the relation between Pe and Pv is expressed by the above-mentioned Equations 1 and 2 (where Ze = f in this 
case). 

[0335] The above-mentioned two equations can detemiine the coordinates, in the plane-of-view coordinate system, 
of each pixel of the image projected onto the plane of view. 

[0336] Procedure 3: Obtain the calculation fonnula for establishing the con^espondence between the plane-of-view 
50 coordinate system and the worid coordinate system. Here, assume that the plane-of-view coordinate system is in the 
following relation with the reference to the worid coordinate system. 

Let the vector from the origin Oe of the plane-of-view coordinate system to the origin Ow of the worid coordinate 
system be (tx, ty, tz). 

55 Thus, the positional shift between the two coordinate systems vanishes by the parallel displacement of the 

plane-of-view coordinate system by (tx, ty, tz). 

When the camera position is Icnown in the worid coordinate system, the (tx, ty, tz) is easily obtained by inverting 
the sign of the coordinates of the position. 
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Let the rotation matrix for aligning the orientation of each axis of the plane-of-'View coordinate system with that of 
the world coordinate system be the above-mentioned Equation (1). 

[0337] This rotation matrix can be easily obtained by obtaining the rotation matrix around each of the X, Y, and Z 
5 axes based on the parameters (a, p, 7) indicating the orientation and the rotation around the camera optica) axis In the 

camera parameter table 102C and by making the composition of them. Here, when a point is represented by Pw (Xw, 

Yw, Zw) in the worid coordinate system and by Pje (Xe. Ye, Ze) in the plane-of-view coordinate system, the relation of 

Equation (8) holds among Pe (Xe. Ye, Ze), Pw (Xw. Yw, Zw). (tx, ty, tz), and the above-mentioned rotation matrix. 

[0338] Usingthese Equations (1), (2), and (8), a pixel Pv(u, v) on the plane of view can be converted to the coordinates 
10 Pw (Xw, Yw, Zw) in the worid coordinate system. 

[0339] The position of each pixel constituting the image shot by the car-carried camera is generally represented by 

the coordinates on a plane including the Image plane. The detail thereof has been described above in Figure 30. 

[0340] Figure 47 shows the example in which X-Z plane is used for the plane to generate the space model 103C. 

[0341] The point-of-view converting means 105C composes an image shot by a camera installed at an arbitrary 
15 virtual point of view from the result of the mapping of the car-carried camera image into the space data 1 03C by the 

mapping means 1 04C. The outline of the method has been descrit)ed above with regard to the above-mentioned point- 

of-vlew converting means 105A. 

[0342] Meanwhile, during the composition of the image viewed from a camera placed at said arbitrary point of view 
by the point-of-view converting means 1 05C. a situation can occur that no color is being mapped in a point of the space 
20 model 103C corresponding to a pixel of said synthesized image. In that case, 

• it may be replaced with a color, such as black, identifiable as a part without an object, or 

• the color of the point in question may be interpolated or extrapolated from the periphery points having a mapped 
color. 

25 

[0343] The camera parameter con'ecting means 1 06C corrects the camera parameters independently for each cam- 
era. 

[0344] Figure 48 is a conceptual diagram showing an example of the configuration of an operating sec^on for cor- 
recting the camera parameters by camera parameter correcting means 106C. 
30 [0345] In the present embodiment, the apparatus comprises: 

• camera selection buttons 901 C for selecting a camera to be corrected; 

• zoom buttons (fonward and backward) 904C for moving forward and backward In the direction of the optical axis 
of the camera; 

35 • parallel displacement buttons (up, down, right, and left) 902C for displacing the camera in parallel to a direction 
perpendicular to the optical axis of the camera; and 

• a joystbk 903C for changing the orientation of the camera and for correcting the rotation around the optical axis 
of the camera; wherein 

40 (a) the three-dimensional position of the camera can be changed by operating the zoom button 904C and the 

parallel displacement button 902C, and 

(b) the rotatbn angle of the camera can be changed by the joystbk 903C, wherein 

the following infonmation of the camera parameters is corrected along with the above-mentioned operatrons (a) 
45 and (b): 

(a) X, Y, and Z coordinates of the camera position in the three-dimensional space coordinate system 

(b) angle a relative to the Y-Z plane, among the camera orientation 

50 angle p relative to the X-Z plane, among the camera orientation 

rotation angle y around the optical axis of the camera. 
[0346] The corrected result is reflected immediately in Equation (8). The rotation matrix shown in Equation (0) is re- 
calculated In Equation (8) using the new camera angle, and the (tc, ty. tz) is replaced by the new camera position. 
Then, an image is synthesized as the result of the correction operation using the altered Equation (8), whereby the 

55 operator can check whether the correction operation by himself is correct or not at a glance. 

[0347] With regard to he operation of the joystbk 903C, at least the following two kinds of operation are Indispensably 
necessary: 
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(1) the operation of moving the rod of the Joystick 903C to up, down, right, and left, which conresponds to the 
adjustment of the direction of the optical axis of the camera; and 

(2) the operation of rotating the rod itself of the joystick 903C. whk:h conresponds to the adjustment of the rotation 
around the optk:al axis of the camera. 

5 

[0348] With regard to (1 ) among these operations of the joystk:k 903C. the operator can easiiy and intuitively grasp 
the direction of the camera, when the operation contents and the result are displayed as if the rod of the joystick S03C 
is Identical to the optk»l axis of the camera. 

[0349] For this purpose, the orientation of the camera is moved in the reversed direction of the movement of rod of 
10 the joystick 903C in the same amount instead that the direction of the moving of the optical axis of the camera is set 
In the same direction as the movement of rod of the joystick 903C. Then, this orientation of the camera Is written into 
the camera parameter table 102C, as the altered orientation of the camera. 

[0350] Further, In order to perfomn such correction as if connecting a rearview mirror, it is preferable to carry out it in 
the place where the camera is actually installed while watching the image of the direction of the camera orientation. 

IS For this purpose, in the present correction process, it is preferable to align the virtual point of view being set for the 
display, with the position and the orientation of the yet-corrected camera to be corrected. As such, the situation that 
the orientation and the position of the camera are changed by the correction can be grasped at a glance. The position 
of the virtual point of view may be restored to the original position on completion of the conrection. 
[0351 ] Figure 49 is a block diagram showing an example of the configuration of an art relevant to the Image generating 

20 apparatus in accordance with the present invention. 

[0352] This configuration, in comparison with the configuration of Rgure 42, further comprises a mapping table 1 08C 
for retaining the correspondence relation between the pixels of a camera Input image and the synthesized image. 
[0353] Figure 50 is a conceptual diagram showing a mapping table 1 08C, In the form of a table. The table is composed 
of the same number of the cells as the number of the pixels of the screen displayed by the displaying means 107C. 

25 That is, the table is configured such that: 

• the number of the columns of the table is the horizontal number of the pixels of the display screen; and that 

• the number of the lines of the table is the vertkxd number of the pixels of the display screen. 

30 Each pixel has the data of: 

• camera number, and 

• pixel coordinates of .the image shot by said camera. 

35 For example, the top left cell of Figure 50 indicates the top left or (0, 0) part of tiie display screen. In response to the 
data contents (1 , 10, 10) stored in said cell, the mapping means 104C performs the process of "displaying the data of 
the pixel (1 0, 1 0) of the image shot by the first camera onto the (0, 0) of the display screen." 
[0354] Meanwhile, in order to generate the mapping table 108C, It is necessary to cateulate the con-espondence 
between the camera input image and the synthesized image. This calculation can be performed easily by the mapping 

40 means 1 04C and tiie point-of-view converting means 1 05C. More specifically, the following processes 1 to 2 are per- 
fomned for all the cameras. 

1 . Select a camera and let its camera number be Cn. 

2. Perfonm the following processes 2.1 to 2.3 for ail the pixels constituting the camera image. 

45 

2.1 The mapping means 1 04C maps the pixel (let the coordinates be the coordinates 1 ) into the space model 
103C. 

2.2 TTie point-of-view converting means 105C obtains the coordinates (coordinates 2) on the plane of view 
when the pixel mapped in 2.1 is viewed from the virtual point of view. 

50 2.3 Write the data of the combination of (Cn, coordinates 1) into the cell conrespondlng to the coordinates 2 

in the mapping table 108C. Here, in case that the display region of each camera is given, the data is written 
in only when the coordinates 2 is within the display region. 

[0355] When the mapping table 1 08C has been generated, the displaying means 107C replaces the camera image 
55 into the display image directly using the table, Instead of displaying the synthesized image using the process result of 
the mapping means 104C and the point-of-vlew converting means 105C. This method permits to speed up the com- 
position process of the image from the plurality of cameras. 

[0356] An embodiment of each means constituting an image generating apparatus in accordance with the present 
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invention has been described above. The flow of the correction process of the image generating apparatus in accord- 
ance with the present invention is described below. 

[0357] Figure 51 is a flow chart showing the flow of the process of correcting the camera parameters. Rgures 52 
and 53 are conceptual diagrams used for the assistance of the description of the process of correcting the camera 
5 parameters. Figures 52{a) and 52(b) show a display screen during the correction of the camera parameters. Rgures 
53(a) and 53(b) show a display screen during the time without the correction of the camera parameters (that is, normal 
state). 

[0358] Process 1 201 : Select a camera to t>e conrecAed using the camera selec^on buttons and the like. In that case, 
a situation is displayed that the synthesized image is shifted as Rgure 53(a). 
10 [0359] Process 1202: Change the virtual point of view con-espondingly to the position and the orientation of the 
selected camera and compose the image. When the virtual point of view is changed, an Image viewed from the point- 
of-view position of the selected camera is displayed on the screen as shown in Figure 52(a), whereby the situation of 
the shift of the camera from the point of view can be grasped. 

[0360] Process 1 203: Con'ec^ the three-dimensional space position and the orientation of the camera using the zoom 
15 buttons 904C, the parallel displacement buttons 902C, and the joystick 903C. The virtual point of view is kept fixed 
during the correction process. After a proper correction, the situation without a shift is displayed on the screen as shown 
in Figure 52(b). 

[0361 ] Process 1 204: On completion of the correction of the camera In question, write the corrected camera param- 
eters Into the camera parameter table 102C. 
20 [0362] Process 1 205: When another camera needs correction, select said camera next and repeat the above-men- 
tioned Processes 1201 to 1204. Othenwise, proceed to the next Process 1206. 

[0363] Process 1206: Restore the virtual point of view changed in Process 1202 into the virtual point of view of a 
time before the correction process of the camera, and compose the image. As a result of the composition, the image 
without a shift is displayed on the screen as shown in Figure 52(b). Checking it, the correction process of the camera 
25 parameters is tenninated. 

[0364] As shown in the conceptual diagrams of Figures 52 and 53, with regard to the part of the border between the 
images by the car-canied cameras, when a marker (for example, a line) which is not so disturiaing to the eye is mariced 
in the border section, it is easily understood which camera is displayed where. 

[0365] As such, in the present invention, an image viewed from an arbitrary virtual point of view is synthesized using 
30 the images from a limited number of cameras. In case that the position and the orientation of the camera are shifted 

due to vibration and the like during the running of the vehicle, said shift has been difficult to correct in the prior art. In 

accordance with the present invention, the driver obtains an optimal image by moving the position and changing the 

orientation of the point of view only for the camera having a shift while checking the displayed image when necessary. 

This process is realized by the camera parameter correcting means for conrecting the point of view, and is carried out 
35 as if a rearview mirror is adjusted. 

[0366] Therefore, even when a shift occurs in a part of the synthesized image, the driver can easily correct the shift 

without a large cost and trouble for the adjustment. 

[0367] Further, in the present invention, the point-of-view converting means changes the virtual point of view into the 
position of the car-canied camera to be corrected thereby to compose the image during the correction operation of the 
40 point of view by the driver. Thus, the correction of the orientation and the position of the car-carried camera can be 
perfomied as if a rearview mln-or is con-ected. 

[0368] Furthermore, in the present invention, the camera parameters, such as the orientation and the position, of 
the point of view being virtually set are corrected instead of correcting the orientation of an actual camera. Thus, the 
mechanism for changing the orientation of the camera becomes unnecessary, whereby the cost of the apparatus can 
45 be reduced. 

[0369] Figure 54 is a block diagram showing another embodiment of the present invention. The same means as that 
of Figure 42 is assigned with the same numeral, and hence the description is omitted. The new point Is to comprise 
guide data storing means 1 1 0C for storing the guide data for guiding the correction of the camera parameters indicating 
the characteristics of a camera. The guide data is superposed onto the input image by the image synthesizing means 
so 109C, and displayed by the displaying means 107C. The guide data can be generated by an arbitrary generating 
method and, for example, Is generated by the following method and used. 

[0370] The configuration of Figure 55 further comprises, in addition to that of Figure 54, characteristics generating 
means 111C for generating a point source of light at a predetermined place of a vehicle and characteristics extracting , 
means 112C capable of extracting the characteristk^s, wherein the characteristk:s extraction result can be stored into 
55 storing means 1 1 0C as said guide data. 

[0371] For example, Figure 56 Is characterized by the point source of light. The front side of the vehicle Is equipped 
with several lamps 113C (a). This lamp 113C corresponds to the characteristk» generating means 111C. Images of 
the front side of the body of the vehk:le shot by the cameras 1 and 2 (f) attached on the front surface Is shown in (b 
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and c). The characteristics extrac^ng means 112C extracts the luminous chara^eristics from the Image by image 
processing (d and e). 

[0372] Figures 57 (a) to (e) show an example in which a line segment of a part of the vehicle is ^en as the char- 
acteristics. That is, the line segment in the rear right comer part of the vehicle is extracted by the image processing. 
5 [0373] The positional relation of these of the point source of light and the line segment is invariant relatively to the 
camera regardless of the movement of the vehicle. Accordingly, when the direction and the like of the camem is changed 
due to vibration, the camera parameters can be corrected using the invariant characteristics. The detailed method for 
this is described below. 

[0374] In Figure 58, a closed circle indicates the image where a point source of light should be present. An open 
10 circle indicates a point source of light shot when the camera is shifted. An image in the shifted state is obtained as 
shown in (a). Then, using the joystick 903C to con^ct the parameters as described above, the circles coincides (b and 
c). As such, the calibration can be performed easily. 

[0375] Figures 59(a) to 59(c) shows an example of the calibration using the above-mentk>ned line segment 
[0376] Here, the above-mentioned characteristtes can be generated by another method such as the irradiation of 
15 pattem light. 

[0377] Another embodiment of the present invention is descn'bed below. 

[0378] Figure 60 is a block diagram showing an example of the configuration of an image generating apparatus in 
accordance with the present invention (an example of Claim 55). 

[0379] An apparatus for monitoring the surroundings of a vehble in accordance with the present embodiment has, 
20 as the basic configuration, the configuration comprising: a plurality of cameras 101 D attached for grasping the situation 
of the surroundings of a vehicle; a camera parameter table 1020 for storing the camera parameters indicating the 
characteristks of cameras 101 D; mapping means 104D for mapping the innages input from the cameras 101 D, into a 
space model 103D generated by modeling the situation of the sunDundings of the vehtele; a point-of-vlew parameter 
table 108D for storing the point-of-view parameters including at least position and direction; potnt-of-view converting 
25 means 1 05D for synthesizing a single image viewed from a desired virtual point of view using the data generated by 
the mapping means 1 04D; point-of-view parameter (X)rre(ting means 1 06D for connecting the parameters of said virtual 
point of view; and displaying means 107D for juncdonlng and displaying the image converted by the point-of-view 
converting means 105D. 

[0380] The detail of each component constituting the present embodiment is described below. 
30 [0381] The camera 1 01 D is a TV camera for shooting an image of a space to be monitored, such as the surroundings 
of a vehicle. The camera Is attached on the vehicle as shown in Figure 43. 

[0382] The camera parameter table 1 02D is a table for storing the camera parameters. The camera parameter is as 
described above. 

[0383] As a preparation for the detailed description of the camera parameter table 1 02D, a three-dimensbnal space 
35 coordinate system based on the vehfcle is defined. This is as described above In Figure 44. 
[0384] The data stored in the camera parameter table 1 02D is the same as that of Figure 45. 
[0385] In the present embodiment, the point-of-view converting means 105D (described later in detail) pemnits to 
convert an image shot by the car-cam'ed camera into an image viewed from a desired virtual point of view. Specif toally, 
the image viewed from the virtual point of view is an Image whteh Is to be seen If a camera is placed at a desired place 
40 and at a desired orientation. Thus, the point-of-view parameters of the virtual point of view can be expressed by the 
same camera parameters as described above. 

[0386] Figure 61 is a diagram showing the data stored in the point-of-view parameter table 108D, in the fomn of a 
table. In Figure 61 , three sets of the point-of-view parameters are stored, and each set corresponds to each of three 
car-cam'ed cameras in one-to-one con^espondence. 
^ [0387] For example, the image shot by the (camera 1 is converted to an image viewed from the virtual point of view 
1 by the point-of-view conversion , and the image shot by the camera 2 is converted to an image viewed from the virtual 
point of view 2 by the point-of-view conversion. 

[0388] As such, by setting a virtual point of view for each camera, when a shit occurs on the display screen in a 
junction part between the image of the camera in question and the image of another camera due to the positional shift 
so and the like of the camera, the changing of the parameters of the virtual point of view only the shifted camera image 
of which is watched pennlts an intuitive correction of the shift. The detail of the method of the confection Is described 
later in the explanatbn of the point-of-view parameter correcting means. 

[0389] The itenns in the second to the ninth columns of the contents desc^be in Figure 61 show an example of the 
point-of-view parameters. It is as follows from left to right. 
55 [0390] That Is, the example of the second line of Figure 61 describes that the virtual point of view 1 is located at a 
coordinates (0, 0, z2), that the orientation is 0 degree relative to the Y-Z plane, -90 degrees relative to the X-Z plane, 
and no rotation around the line-of-sight axis, that the focal length Is f2, and that the lens distortion coefficients k1 and 
k2 are both zero. 
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[0391] The mapping means 104D maps each pixel constituting the Input image from the car-carried camera, into a 
space model 103D depending on the camera parameters. That is. it pefspectively projects each image shot by the 
car-canied camera into the space model 1 03D. 

[0392] l-lere, the space model 1 03D is a three-dimensional model where the mapping means 1 04D maps the image 
5 from the camera into the three-dimensional space coordinate system. The model used is, for example, a model con- 
sisting of a plane, a curved surface, or a plane and a curved surface. In the present embodiment, for the simplicity of 
description, a plane model of the road surface is used as an example of the simplest space model 103D. The method 
of mapping the image by the car-carried camera into said plane model is described below. 

[0393] in pra(^ice, such a simple model causes a problem, for example, that an object having a height component 
10 is substantially deformed. Thus, a space model consisting of several planes, a screen plane consisting of planes and 
curved surfaces, and the like are more preferable. Further, when an accurate three-dimensional model of the obstacles 
in the surroundings of the vehicle can be obtained in real time, an synthesized image with higher precision is obtainable 
using such a three-dimensional model. 

[0394] Before the description of the mapping process by the mapping means 104C, the method for converting the 
15 plane-of-view coordinates to the wo rid coordinates is necessary to describe, however, this is the same as described 
above with regard to Figure 47. 

[0395] The position of each pixel constituting the Image shot by the car-canied camera is generally represented by 
the coordinates on a plane including the image plane. The detail thereof has been described above in Rgure 7 of '233. 
[0396] The polnt-of-view converting means 105D composes an image shot by a camera installed at an arbitrary 
20 virtual point of view from the result of the mapping of the car-carried camera image into the space data 1 03D by the 
mapping means 1 04D. The outline of the method has been described above with regard to the point-of-view converting 
means 105A. 

[0397] Meanwhile, during the composition of the image viewed from a camera placed at said arbitrary point of view 
by the point-of-view converting means 1 05D, a situation can occur that no color is being mapped in a point of the space 

25 model 1 03D corresponding to a pixel of said synthesized image. That case has been described above. 

[0398] Figure 62 is a conceptual diagram showing the situation that each pixel constituting the input image from a 
car-carried camera is mapped onto a plane of the space model 1 03D by the mapping means 1 04D and that an image 
shot by a camera installed at the virtual point of view is synthesized from the result mapped onto the plane. 
[0399] The point-of-view parameter con-ecting means 1 06D can correct the parameters of the virtual point of view 

30 being set for each camera, independently for each camera. 

[0400] Figure 63 is a conceptual diagram showing an example of the configuration of an operating section for cor- 
recting the virtual-point-of-view parameters by point-of-view parameter correcting means 1 06D. 
[0401 ] In the present embodiment, the apparatus comprises: 

35 • point-of-view selection buttons 1 001 for selecting a virtual. point of view to be connected; 

• zoom buttons (f onward and backward) 1 004 for moving forward and backward on the center axis of the line of sight 
from the virtual point of view; 

• parallel displacement buttons (up, down, right, and left relative to an image plane) 1002 for displacing the virtual 
point of view in parallel to a direction perpendicular to the line of sight from the point of view; and 

40 • a joystick 1 003D for changing the direction of the line of sight from the virtual point of view and for correcting the 
rotation around the direction of the line of sight from the virtual point of view; wherein 

(a) the three-dimensional position of the virtual point of view can be changed by operating the zoom button 1 004D 
and the parallel displacement button 1 002D. and 

45 (b) the direction of the line of sight from the virtual point of view can be changed by the joystbk 1 003D, wherein 

the following infomnation of the virtuat-point-of-view parameters in question is connected and overwritten on the 
point-of-view parameter table 108D along with the above-mentioned operations (a) and (b): 

so (a) X. Y. and Z coordinates of the virtual-point-of-view position in the three-dimensional space coordinate system 

(b) angle a relative to the Y-Z plane, among the direction of the line of sight from the virtual point of view 

angle p relative to the X-Z plane, among the directron of the line of sight from the virtual point of view 
rotation angle y around the center axis of the line of sight from the virtual point of view. 
55 [0402] The corrected result is reflected imnnediately in Equation (8). The rotation matrix shown in Equation (0) is re- 
calculated in Equation (8) using the new direction of the line of sight, and the (tx, 1y, tz) is replaced by the coordinates 
of the new virtual-point-of-view position. 

[0403] Then, an image is synthesized as the result of the correction operation using the altered Equation (8). By 
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watching the synthesized image, the operator can checic whether the correction operation by himself is correct or not 
at a glance. 

[0404] Among the operation of the parameter correction of the virtual point of view, the operation of changing the 
orientation, the position, and the rotation of the virtual point of view is intuitively understandable for the operator when 
5 the contents of the operation agrees with the behavior of the onage of the operation result For this purpose, in the 
following three operations among the operation of the parameter conrection, the operation directjon of the operation of 
changing the orientation, the position, and the rotation of the virtual point of view is set reverse to the actual change 
direction of the point-of-view parameters, as is described in Claim 57 of the present invention: 

10 (1) the operation of moving the rod of the joystick 1003D to up, down, right, and left, which corresponds to the 

adjustment of the direc^on of the optical axis of the camera; 

(2) the operation of rotating the rod itself of the joystick 1 003D, whk:h con^esponds to the adjustment of the rotation 
around the optk:al axis of the camera; and 

(3) the operation of the parallel displacement buttons 1 002D, whfch corresponds to the operation of disptadng the 
15 virtual point of view In parallel to a direction perpendteular to the line of sight from the point of view. 

[0405] That is, with regard to the joystk:k 1 003D for example, the direc^on of the line of sight is moved in the reversed 
direction of the movement of rod of the Joystick 1003D in the same amount instead that the direction of the moving of 
the direction of the line of sight is set in the same direction as the movement of rod of the joystick 1003D. Then, the 

20 moved state is written into the point-of-view parameter table 102D. as the altered direction of the line of sight. For 
example, when the joystfek 1 003D is tilted to right by 1 0 degrees, the point-of-view parameter is then set to a value in 
whteh the directton of the line of sight is tilted to left by 1 0 degrees relatively to the current line of sight. Similarly, when 
the joystick 1003D is rotated clockwise by 5 degrees, the point-of-view parameter is then set to a value in which the 
rotation angle around the center axis of the line of sight is rotated anticlockwise by 5 degrees. This situation is the 

25 same also for the operation of the parallel displacement button 1 004D. 

[0406] Figure64 is a block diagram showing an example of the configuration of an art relevant to the image generating 
apparatus in accordance with the present invention:^ 

[0407] This configuration, in comparison with the configuration of Figure 60, further comprises a mapping table 1 09D 
for retaining the correspondence relation between the pixels of a camera input image and the synthesized image. 
30 [0408] A conceptual diagram showing a mapping table 1 09D in the form of a table has the same contents as Rgure 
50, and hence the detail is omitted. The table is composed of the same number of the cells as the number of the pixels 
of the screen displayed by the displaying means 107D. 

[0409] In such an image generating apparatus, when the point-of-view parameter is conrected by the point-of-view 
parameter connecting means, the part, in whteh an alteratbn occurs due to the correction, of the con^espondence relation 
35 between the pixels of the camera Input image and the synthesized image is reK^alculated by the above-mentioned 
procedure. 

[041 0] When the mapping table 1 09D has been generated, the displaying means 1 07D replaces the camera image 
into the display image directly using the table, instead of displaying the synthesized image using the process result of 
the mapping means 104D and the point-of-view converting means 105D. This method pemilts to speed up the com- 

40 position process of the image from the plurality of cameras. 

[041 1 ] Another method is described below for the fast correction of the mapping table in case that the position, the 
orientation, and the like of the virtual point of view is changed by said point-of-view parameter coaecting means. 
[0412] Figure 65(a) is a block diagram showing an example of the configuration of an art relevant to an image gen- 
erating apparatus in accordance with the present invention. Rgure 65(b) is a conceptual diagram showing an example 

45 of a car-canied camera synthesized image before the point-of-view parameter correction. Figure 65(c) is a conceptual 
diagram showing an example of a car-carried camera synthesized image after the point-of-view parameter correction. 
[0413] The present image generating apparatus has the configuration, in addition to the previous apparatus, further 
comprising: mapping table correcting means 11 OD for re-caknjlating said mapping table using the point-of-view pa- 
rameter altered by the process of the point-of-view parameter con'ecting means; and a buffer 111 D for temporarily 

50 storing the data being under tiie process of the mapping table correcting means 11 OD. 

[0414] The mapping table conrecting means 110D can alter the contents of the above-mentioned mapping table 
without performing the complicated cafculation of Equation (8), by reflecting the conrection result of the point-of-view 
parameter into Equation (8). The method of altering the mapping table Is described below sequentially 
[0415] Figures 66(a) to 66(c) are conceptual diagrams used for the assistance of the descriptbn of the method of 

55 altering the mapping table. In these figures, a gray j3arallelogram Indicates the image plane before the virtual-point-of- 
view correction, and a white parallelogram indicates the image plane after the virtual-point-ot-view correction. The 
configuration of the operating section for the connection of the virtual-point-of-view parameters is assumed to be as 
shown in Rgure 63. 
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Process 1 : Prepare previously the buffer 111 D for temporarily storing the alteration result of the mapping table. 
Process 2: Obtain the point-of-view parameter connection information of any of parallel displacement, direction, 
rotation, and zoom by the point-of-view parameter correcting means. Calculate the data (table conversion data 
hereafter) of the correspondence relation between the display coordinates (u, v) before the virtual-point-of-view 
5 con-ection and the display coordinates (u', v^ after the con-ection, by any of the following four processes corre- 

spondingly to the contents of the infonmation, and then write it into the buffer 111D. 



(1) The operation of the parallel displacement button 

10 [0416] In case of parallel displacement, acquire the displacement distance (du, dv) before and after the displacement 
in the unit of pixel from the parallel displacement button as shown In Rgure 66(a). Obtain the relation between the 
coordinates PI before the alteration and the coordinates PI' after the alteration using Equation (10), and then write 
the above-mentioned relation into the buffer 1 1 1 D. 

Pi:(Ui.V,)^P,:(u'i.vV 

= (u^ - du. v^ - dv) Equation 10 



20 (2) The operation of the direction correction by the joystick 

[0417] In case of the direction of the line of sight from the virtual point of view, acquire the difference (e, ^) between 
the angles before and after the alteration as shown In Figure 66lb). Obtain the relation between the coordinates P2 
before the alteration and the coordinates P2' after the alteration using Equations (11) and (12), and then write the 
25 above-mentioned relation Into the buffer 1 1 1 D. 



u-2= "-d-tone Equation 11 

d 



v'g= "-d>tan<|> Equation 12 

1 + ^ . tan <|> 

35 ^ 
(3) The operation of rotation by the joysticic 

[0418] In case of the correction of rotation around the direction of the line of sight from the virtual point of view, 
^ acquire the rotation angle p before and after the atteration as shown in Figure 66(c). Obtain the relation between the 
coordinates PS before the alteration and the coordinates P3' after the alteration using Equation (13), and then write 
the above-mentioned relation into the buffer 11 ID. 



45 



Equation 13 P a : ( u 3, V 3)-^P'3 : ( u 3, v'3) 



= (U3V3)f cosps lup 
50 V~sinpco sp 



(4) The operation of the zoom button 

55 

[0419] In case of the correction of zoom (magnification), acquire the magnification k before and after the alteration 
from the zoom button. Obtain the relation between the coordinates P4 before the alteration and the coordinates P4' 
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after the alteration using Equation (1 4), and then write the above-mentioned relation into the buffer 1 1 1 D. 

^ = (Ic • U4,lc • U4) Equation 14 

Process 3: A!ter the mapping table by refening to the buffer 111 D. The contents of the atteration of the coordinates 1 
of each pixel of the mapping table is written by the above-mentioned Process 2, for example, as the following table 
10 conversion data in the buffer 1110. 

(coordinates before alteration: P, coordinates after alteration: P) 
[0420] By referring to the data, when all the cell in the mapping table satisfy the following conditions (a) and (b), alter 
the coordinates 1 : P before alteration of the cell into the coordinates P after alteration of said table conversion data. 

^5 (a) The camera number of the cell coincides with the number of the car-carried camera corresponding to the virtual 

point of view which is the objective of the correction. 

(b) The coordinates 1 : P before alteration of the cell coincides with the coordinates of the left side temi of said 
table conversion data. 

20 [0421] Here, the cell in which the camera number of the cell coincides with the number of the car-canied camera 
corresponding to the virtual point of view which is the objective of the correction but in which the correction of the 
coordinates 1 is not perfonmed is concluded to be out side of the field of view from the virtual point of view in which the 
correction has been performed. Jhus, coordinate values of the region outside of the display area of the displaying 
means are written in. In that case, the displaying means can display a color, such as blacic, indicating the out-field 
when undisplayable coordinates are input. 

[0422] An embodiment of each means constituting an image generating apparatus in accordance with the present 
invention has been described above. The flow of the correction process of the image generating apparatus in accord- 
ance with the present invention is described below. 

[0423] Figure 67 is a flow chart showing the flow of the process of con-ecting the point-of-view parameters in the 
^ image generating apparatus of the present inveniton. Figures 68(a) to 68(c) are conceptual diagrams used for the 
assistance of the description of the process of correcting the point-of-view parameters. 

1 . (Process 1501) Select a virtual point of view to be connected using the point-of-view selection buttons 1001D 
and the like. In that case, a situation is displayed that the syntitesized Image is shifted as Figure 68(a). 
^ 2. (Process 1502) As described in Claim 58 of the present invention, replace the virtual point of view In c^n^ection 

by a tentative virtual point of view (for example, a point of view before connection) and then fix it. 

3. (Process 1 503) Correct the three-dimensional space position of the virtual point of view and the direction of the 
line of sight using the zoom buttons 1004D, the parallel displacement buttons 1002D, and the joysticlc 1003D. 
Since the point of view is fixed by the above-mentioned Process 2 during the correction process, the woric of the 

^ point-of-view correction can be carried out as if a camera placed at the point of view is adjusted by hand. If the 

con-ection is proper, the situation without a shift is displayed, for example, as shown in Figure 68(c), after the 
screen shown in Figure 68(b) is displayed. 

4. (Process 1 504) On completion of the comection of the virtual point of view In question, write the corrected point- 
of-view parameters into the point-of-view parameter table 1 08D. 

^ 5, (Process 1505) When another camera needs correction, select said camera next and repeat the above-men- 

tioned Processes 1501 to 1504. Otherwise, proceed to the next Process 1506. 

6. (Process 1506) Release tiie state that the virtual point of view Is fixed to the tentative virtual point of view. As a 
result of tiie composition, the situation without a shift Is displayed on the screen as shown in Figure 52(b). Checking 
it, tiie correction process of the point-of-view parameters is tenminated. 

50 ... 

[0424] As shown in the conceptual diagranr^ of Figures 68(a) to 68(c), with regard to the part of the border between 
the images by the c:ar-carried cameras, when a maricer (for example, a broken line) whicti is not so disturbing to the 
eye is mariced in the border section, It is easily understood whtoh camera is displayed where. 
[0425] As such, in the present invention , an image viewed from an artsitrary virtual point of view is synthesized using 
the images from a limited number of cameras. In case that the position and the orientation of the camera are shifted 
due to vibration and the like during the running of the vehicle, the woric to align the shifted image with the other image 
has been difficult in the prior art because of the prior art uses the method such that the shift is restored by the fine 
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adjustment of the camera itself. However. In the present invention, by introducing the concept of virtual point of view, 
the image having a shift can be solely corrected by the potnt-of-view parameter correcting means while watching the 
Image viewed from the virtual point of view as if a camera placed at the point of view Is adjusted. At that time, the 
corrected state of the movement and the orientation of the point-of-view position are fed back successively as an 

5 image. Accordingly, the woricer of the correction can obtain the optimal image by checking the display image suoces- 
shrely. Therefore, the work of the alignment of the Images becomes substantially easy in comparison with the prior art, 
and a small shift does not require the adjustment of the camera parameters using a large apparatus. 
[0426] Further, in the present inventton, the camera parameters, such as the orientation and the position, of the point 
of view being virtually set are corrected Instead of correc^ng the orientation of an actual camera. Thus, the mechanism 

10 for changing the orientation of the camera becomes unnecessary, whereby the cost of the apparatus can be reduced. 

Industrial Applicability 

[0427] An Image generating apparatus of the present Invention is applicable in the fields of the monitoring of the 
IS surroundings of a vehicle, the monitoring of the inside of a shop, and the like, and capable of easily synthesizing a 
synthesized image having the feeling as if actually viewed from a virtual point of view, using the Images shot.by re- 
spective cameras. 

[0428] Further the apparatus has a f eatu re that a shift in the direction of a camera can be easily corrected in the image. 

20 

Claims 

1 . An Image generating apparatus comprising: a camera or a plurality of cameras; space reconstructing means for 
mapping the input Image from said camera, Into a predetemnined space model of a predetennlned three-dimen- 
25 sional space; point-of-vlew converting means for synthesizing an Image viewed from an arbitrary virtual point of 

view in said predetermined three-dimensional space by referencing to said space data mapped by said space 
reconstructing means; and displaying nneans for displaying the image converted by said polnt-of-view converting 
means. 

30 2. An image generating apparatus of Claim 1 comprising a camera parameter table for storing a camera parameter 
Indicating the characteristics of said camera, wherein said space reconstructing means performs the mapping 
depending on said camera parameter. 

3. An image generating apparatus of Claim 1 wherein said space model is a predetermined plane in said three- 
35 dimensional space. 

4. An image generating apparatus of Claim 3 wherein said camera is attached on a vehble and said predetemnlned 
1 plane is the road surface under the vehicle. 

40 5. An Image generating apparatus of Claim 1 wherein, during the constitution of a predetermined point P of said 
space model as a pixel from said virtual point of view, in case that said point P Is corresponded to an image shot 
by a only camera, said point-of-view converting means determines the color of said pixel using the color of the 
pixels of the Image from tiie camera. 

45 6. An Image generating apparatus of Claim 1 wherein, during the constitution of a predetermined point P of said 
space model as a pixel from said virtual point of view, in case that said point P is corresponded to an Image shot 
by said plurality of cameras, said point-of-view converting means determines the color of said pixel using the color 
of the pixels of the images from those plurality of camera. 

An image generating apparatus of Claim 1 wherein, during the constitution of a predetemnlned point P of said 
space model as a pixel from said virtual point of view, in case that said point P is not corresponded to an image 
shot by any camera, said point-of-view converting means determines the color of said pixel using a color interpo- 
lated by the color of the periphery of said point P or using such a color that can be identifiable as a part where 
there is no object. 

An Image generating apparatus of Claim 2 comprising calibrating means for obtaining a camera parameter indi- 
cating said camera characteristtes by Input or cateulation. 
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9. An image generating apparatus of Claim 8 comprising characteristic point extracting means for acquiring, by man- 
ual input or automatic extraction, a plurality of characteristic points necessary to obtain a camera parameter by 
said calculation, wherein 

said calibrating means calculates at least any of the attachment position of the camera, the attachment angle 
5 of the camera, the lens distortion correction value of the camera, and the focal length of the camera among various 

camera parameters using the characteristic point extracted by said characteristic point extracting means. 

1 0. An image generating apparatus of Claim 9 wherein said characteristic point extracting means includes the function 
of extracting a plurality of characteristic points the coordinate value of which is known in said three-dimensional 

10 space, and wherein 

said calibrating means calculates said camera parameter using said plurality of characteristic points. 

1 1 . An Image generating apparatus of Claim 1 0 wherein said camera is attached on a vehicle and wherein a maricer 
provided In a pari of the vehicle is used as the plurality of characteristic points the coordinate value of which is 

15 known in said three-dimensional space. 

12. An image generating apparatus of Claim 1 0 comprising characteristic point generating means for generating, within 
the field of view of said camera, a plurality of characteristk: points the coordinate value of which is known In said 
three-dimensional space. 

20 

13. An image generating apparatus of Claim 12 wherein said characteristic point generating means has the function 
of irradiating a pattern light as said characteristic point within the field of view of said camera. 

14. An image generating apparatus of any of Claims 8 to 13 comprising a temperature sensor for measuring the 
25 temperature and a temperature correction table for storing the amount of the variation in said camera parameter 

which changes in response to the temperature, wherein 

said calibrating means has the function of monitoring the temperature of said temperature sensor succ^es- 
sively and of correcting said camera parameter depending on sakj temperature correction table in response to the 
temperature ctiange. 

30 

15. An image generating apparatus of Claim 14 wherein said temperature sensor is attached In the vicinity of or in 
association with each of said camera. 

1 6. An image generating apparatus of any of Claims 1 , 2, 3, and 5 to 1 0, and 1 2 to 1 5, wherein said camera is attached 
35 on a vehtete. 

1 7. An image generating apparatus of any of Claims 4, 1 1 , and 1 6, wherein said space data generated by said space 
rec:onstructing means is temporarily stored in a space data buffer, and wherein said apparatus conrtprises: move- 
ment direction detecting means for detecting the movement direction of said vehicle; movement distance detecting 

40 means for detecting the movement distance of the vehicle in a unit time; and space data converting means for 

converting the space data stored in said space data buffer using both the movement direction of the vehicle detected 
by said movement direction detecting means and the movement distanc^e of the vehicle detected by said movement 
distance detecting means. 

^ 18. An image generating apparatus of Claim 1 wherein said camera is attached on a vehicle, and wherein said appa- 
ratus comprises: camera connection indicating means for detecting a situation requiring the calibration of the camera 
thereby indicating a camera calibration when the camera calibration is necessary; and correction history recording 
means for recording the date and time or the running distance when said camera calibration is perfomned. 

so 19. An image generating apparatus of Claim 18 wherein said camera correction Indteating means has the function of 
referencing to said con-ection history reconJing means thereby indicating a camera calibration to the driver when 
a predetennined time has been elapsed since the time of the previous calibration or when the running distance 
since the time of the previous calibration has reached at a predetermined distance. 

55 20. A camera calibration indicating apparatus comprising: correction history rec^ording means for recording the date 
and time or the running distance when said camera calibration is perionned; and camera connection indicating 
means for referencing to said c^orrection history recording means thereby indicating a camera calibration to the 
driver when a predetennined time has been elapsed since the time of the previous calibration or when the running 
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distance since the time of the previous cailbration has reached at a predetemnined distance. 

21 . A camera calibrating apparatus comprising: at lest one camera; a temperature sensor for measuring the temper- 
ature; and calibrating means for detennining the camera parameter indicating the camera characteristics depend- 

5 ing on the temperature measured by said temperature sensor. 

22. A camera calibrating apparatus of Claim 20 wherein said calibrating means comprises: a camera parameter table 
for storing said camera parameter, and a temperature correction table for storing the amount of the variation in 
said camera parameter which changes in response to the temperature change; and refers to said temperature 

10 correction table in response to the temperature change thereby altering said camera parameter. 

23. A camera calibrating apparatus comprising: at least one camera; characteristic point generating means for gen- 
erating, within the field of view of said camera, a plurality of characteristic points the coordinate value of which is 
known in the three-dimensional space where said camera locates; characteristic point extracting means for ex- 

15 tracting said plurality of characteristic points; and calibrating means for calculating the camera parameter indicating 

said carrxBTB characteristics using the plurality of characteristic points extracted. 

24. A camera calibrating apparatus of Claim 23 wherein said ctiaracteristic point generating means has the function 
of Irradiating a light pattem as said characteristic point within the field of view of said camera. 

20 

25. An image generating method comprising: a space reconstructing step of mapping the input image from a camera 
locating in a predetemnined three-dimensional space, into a predetermined spaco model of said predetennined 
three-dimensional space; and a point-of-view converting step of synthesizing an image viewed from an arbitrary 
virtual point of view in said predetermined three-dimensional space by referencing to said space data. 

25 

26. An image generating method of Claim 16 wherein said space model is a plane constituting a part of said three- 
dimensional space. 

27. An image generating method of Claim 25 wherein said mapping is performed depending on the camera parameter 
30 indicating said camera characteristics. 

28. An image generating method of Claim 27 comprising a calibrating step of obtaining a c^amera parameter indicating 
said c^amera characteristics by Input or calculation. 

315 29. An image generating method of Claim 28 wherein said calibrating step Is a step of obtaining the c^amera parameter 
by c:alculation, and wherein said method comprises a characteristic point extracting step of acquiring, by manual 
input or automatic extraction, a plurality of characteristic points necessary for the calculation of said camera pa- 
rameter. 

40 30. An image generating method of Claim 29 comprising a characteristic point generating step of generating, within 
the field of view of said camera, a plurality of characteristic points the coordinate value of which is known in said 
three-dimensional spaco. 

31. An image generating method of Claim 30 wherein said characteristte point generating step includes at least the 
45 step of inBdlating a light pattem as said characteristic point within the field of view of said camera. 

32. An image generating method of Claim 25 wherein said camera is attached on a vehk:le, and wherein 

said method comprises: a movement direction detecting step of detecting the movement direction of said 
vehicle; a movement distance detecting step of detecting the movement distance of said vehicle in a unit time; 
so and a space data converting step of converting the space data using both the movement direction of the vehicle 
detected by said movement direction detecting means and the movement distance of the vehicle detected by said 
movement distanco detecting means. 

33. An image generating method of Claim 25 wherein said camera is attached on a vehk:le, and wherein 

55 said method comprises: a camera correctton indbating step of detecting a situation requiring the calibration 

of said camera thereby indicating a camera calibration when the camera calibration Is necessary; and a con-ection 
history recording step of recording the date and time or the mnning distance when the camera calibration is per- 
formed. 
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34. An image generating apparatus of Claim 1 wherein said camera is attached on a vehicle and said space recon- 
structing means comprises space model generating means for generating said space model. 

35. An image generating apparatus of Claim 1 wherein said space model is a model of one or more planes, a nnodel 
5 of one or more curved surfaces, or a model of the combination of one or more planes and one or more curved 

surfaces. 

36. An image generating apparatus of Claim 1 wherein said space model generating means generates, as a space 
model, at least a first plane of the road surface and at least one second planes intersecting said first plane. 

10 

37. An imagegeneratingapparatusof Claim 36 comprising: a distance sensor for measuring the distance; and obstacle 
detecting means for measuring, as the situation of the surroundings of a veh icie, at least the distance to an obstacle 
existing In the sunx>undings of the vehicle by said distance sensor, and wherein 

said second plane generated by said space model generating means is generated when the distance from 
IS the vehicle to the obstacle detected by said obstacle detecting means is smaller than a predetemiined value. 

38. An image generating apparatus of Claim 37 wherein said second plane generated by said space model generating 
means is removed from said space model when said distance from the vehicle to the obstacle is larger than said 
predetennlned value. 

20 

39. An image generating apparatus of Claim 34 wherein said space model generating means generates a space model 
in response to the situation of the surroundings of said vehicle. 

40. An image generating apparatus of Claim 39 comprising: a distance sensor for measuring the distance; and obstacle 
25 detecting means for measuring, as the situation of the sun^oundings of said vehicle, at least the distance to an 

ot>stacle existing in the surroundings of the vehicle by said distance sensor. 

41 . An image generating apparatus of Claim 39 wherein said vehicle situation is the condition of the road surface in 
the surroundings of said vehicle, and wherein said apparatus comprises road-surface characteristics deleting 

30 means for detecting the characteristics of the condition of the road surface. 

42. An image generating apparatus of Claim 41 wherein a white line indicating a parking space, an end point of the 
white line, or a comer of the intersecting white lines Is used as said characteristics of the condition of the road 
surface. 

35 

43. An image generating apparatus of Claim 39 further comprising: movement direction detecting means for detecting 
the movement direction of said vehicle; and movement distance detecting means for detecting the movement 
distance of the vehicle in a unit time; and wherein said apparatus successively correct said space model depending 
on the present position of the characteristics on the road surface obtained from the process results of said move- 

^ ment direction detecting means and said movement distance detecting means. 

44. An image generating apparatus of Claim 39 or 41 comprising characteristics correcting means for correcting the 
process result while displaying the process result of said road-surface characteristics detecting means on said 
displaying means. 

45 

45. An image generating apparatus comprising: a camera or a plurality of cameras; a camera parameter table for 
storing the camera parameter indicating said camera characteristics; mapping means for mapping the input image 
from said camera, into a predetermined space model of the three-dimensional space using said camera parameter; 
point-of-view converting means for synthesizing an image viewed from each of said plurality of virtual points of 

50 view In said predetemnined three-dimensional space by referencing to said space data mapped by said mapping 
means; and camera paranneter correcting means for correcting the parameter of said camera. 

46. An image generating apparatus of Claim 45 wherein said point-of^view converting means switches said virtual 
point of view according to the time of process or the time of nonrial operation of said camera parameter correcting 

55 means. 

47. An image generating apparatus of Claim 46 wherein said point-of-view converting means determines said virtual 
point of view depending on the camera parameter of any of said cameras in the tinne of process of said camera 
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parameter correcting means. 

48. An image generating apparatus of Claim 47 wherein said point-of-view converting means determines said virtual 
point of view depending on the yet-corrected camera parameter of the camera being under the correction, in the 

5 time of process of said camera parameter correcting means. 

49. An image generating apparatus of any of Claims 45 to 48 wherein the direc^on of the operation of changing the 
direction of the point of view is set to be reverse to the direction of said point of view, in said point-of-view converting 
means. 

10 

50. An image generating apparatus of any of Claims 45 to 49 wherein said displaying means displays a mark indicating 
a boundary, in superposition on the synthesized image in the boundary part where respective images contact when 
displaying the image from each camera. 

IS 51. An image generating apparatus of Claim 45 comprteing: guide data recording means for storing the guide data 
serving as a guide for the correction of said camera parameter; and displaying means for displays the input image 
from said camera and said guide data in superposition. 

52. An image generating apparatus of any of Claims 45 to 51 wherein said camera is attached on a vehicle. 

20 

53. An image generating apparatus of Qaim 52 wherein said guide data is the characteristics of the body of said 
vehicle represented by a point and/or a line segment. 

54. An image generating apparatus of Claim 52 wherein said guide data is obtained using: characteristic point gener- 
25 ating means for generating, within the field of view of said camera, a plurality of characteristic points the coordinate 

value of which is known in the three-dimensional space; characteristic point extracting means for extracting said 
plurality of characteristic points; and said plurality of extracted characteristic points. 

55. An image generating apparatus comprising: a camera or a plurality of cameras; image inputting means for inputting 
30 an image from said camera; a camera parameter table for storing the camera parameter indicating said camera 

characteristics; mapping means for mapping, using said camera parameter, each pixel constituting the input image 
into a predetermined space model of a predetennined three-dimensional space generated by the modeling of the 
situation of the surroundings of the vehicle; a point-of-view parameter table for storing the virtual-point-of-view 
parameter including at least the data of the position and the orientation for each of a plurality of points of view; 
35 point-of-view converting means for synthesizing an image viewed from each of said plurality of virtual points of 

view of said point-of-view parameter table in said predetermined ttiree-dimensional space by referencing to the 
space data mapped by said mapping means; point-of-view parametercorrecting means for correcting the contents 
of said point-of-view parameter table; and displaying means for displaying the image converted by said point-of- 
view converting means. 

40 

56. An image generating apparatus of Claim 55 wherein said plurality of virtual points of view are corresponding to 
said cameras, respectively. 

57. An image generating apparatus of Claim 55 or 56 wherein the direction of operation is set to be reverse to the 
45 actual alteration of point-of-view parameter at least for the operations in orientation, position, and rotation among 

the operations of changing said virtual-point-of-view parameter in said point-of-view parameter correcting means, 

58. An image generating apparatus of any of Claims 55 to 57 wherein, when said point-of-view parameter connecting 
means corrects said virtual-point-of-view parameter, a fixed tentative virtual point of view is provided, and the 

so progress of the correction of the virtual point of view being under the correction is successh^ely displayed as a 

synthesized image viewed from said tentative virtual point of view. 

59. An image generating apparatus of any of Claims 55 to 58 wherein said displaying means displays a marie Indicating 
a boundary, in superposition on the synthesized image in the boundary part where respective images contact when 

55 displaying the image from eacti camera. 

60. An image generating apparatus of any of Claims 55 to 59 wherein said camera is attached on a vehicle. 
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